Method and device for recognizing lines in a camera image for a machine or transmission system, and recognition system for a machine or transmission system

Abstract

The technique presented herein relates to a method for recognizing a line (110) in a camera image for a machine (100). The method includes a reading step, a recognition step, and a generation step. In the reading step, a camera signal (140) is read from an interface for a camera (120), the camera signal (140) representing a camera image of the camera (120). In the recognition step, the presence of a line (110) in the camera image is recognized using at least one anchor (145) having at least one base anchor point (150). In the generation step, a linear regression (155) of the line (110) is generated using the at least one base anchor point (150), the linear regression (155) having a first regression point shifted relative to the base anchor point (150) to recognize the line (110) and at least one further regression point generated relative to a previously generated regression point.

Description

[Technical Field]

[0001] The method is based on an apparatus or method in the technical field to which the independent claims belong. Computer programs are also within the scope of the method. [Background technology]

[0002] Driver assistance systems use lane recognition, for example, for lane assist. Summary of the Invention

[0003] Against this background, the present application proposes a method for recognizing lines in a camera image for a machine or transmission system, as well as an apparatus using this method, a corresponding computer program, and finally a recognition system for a machine or transmission system, according to the independent claims. Advantageous developments and improvements of the method defined in the independent claims are possible thanks to the measures set out in the dependent claims.

[0004] An advantage achievable by the presented approach is that it allows a particularly uniform prediction of the lines. A method for recognizing a line in a camera image for a machine or transmission system is presented. The method includes a reading step, a recognition step, and a generation step. In the reading step, a camera signal is read from an interface for the camera, the camera signal representing a camera image of the camera. In the recognition step, the presence of a line in the camera image is recognized using at least one anchor, and according to one embodiment, multiple anchors each having at least one base anchor point. In the generation step, a linear regression of the line is generated using the at least one base anchor point, the linear regression having a first regression point shifted relative to the base anchor point and at least one further regression point generated relative to a previously generated regression point to recognize the line.

[0005] The method can be implemented, for example, in software or hardware or a mixture of software and hardware, for example in a controller. The machine may be a fully or partially computer-controlled machine, such as a vehicle. The transmission system may be a system for transmitting information, such as a surveillance system or a medical system, e.g., an imaging system. The camera may be a vehicle camera, such as a vehicle camera of an environment capture system for a vehicle. In the recognition step, for example, a grid may be overlaid on the camera image, and the presence of a line may be recognized when a line is recognized in at least one cell of the grid. In the recognition step, for example, a feature extractor may be used to perform classification of the recognized object to recognize the line. The line may represent a road marking or an edge of an object. For example, the line may represent part of a lane. The anchor may have a template or pattern that is searched for in the camera image. For example, different anchors may be used to recognize different lines in the camera image. In the generation step, a linear regression may be generated to correct for displacement (also called "offset"). For example, in the generation step, a linear regression may be generated when base anchor points are recognized at cell centers of cells of the grid and / or on lines. For example, the generating step can generate a linear regression, where a first regression point is shifted on the x-axis relative to a base anchor point to recognize a line, a first additional regression point is generated by shifting the first regression point on the x-axis and on a y-axis extending perpendicular to the x-axis, and subsequent additional regression points are generated by shifting the x-axis and y-axis relative to the previously generated regression point, respectively. Such a method enables linear regression, which may be called a "regression representation," for lane recognition by predicting the offset / deviation of each point along a line / lane relative to the previous point, rather than relative to the anchor. This advantageously results in a particularly uniform prediction of the line. The prediction represents the displacement in image coordinates that needs to be applied to the anchor to fit the line to the image.

[0006] The method may further comprise an output step in which the linear regression is used to output a control signal for controlling a function of a machine, in particular a vehicle, or a transmission system, so that the linear regression, which can predict the course of lines, for example in the form of lanes, can be used to control a vehicle, for example to keep the vehicle within the lane.

[0007] According to one embodiment, in the recognition step, the line may be recognized in an edge region of the camera image using at least one anchor, and / or may be recognized in a cell of the camera image divided into a plurality of cells using an anchor for each cell. For example, in the recognition step, the line may be recognized at the cell center of the cell. Therefore, the line can be quickly and easily found in the edge region of the camera image in each cell.

[0008] According to an embodiment, it is further advantageous if in the recognition step the line in the camera image is recognized when at least one base anchor point of the anchor passes through the line. However, in the recognition step it is also possible to recognize the line in the camera image when only the base anchor point passes through or passes close to the line. Therefore, an anchorless method can be used to recognize the line, and only the base anchor point is needed as the anchor.

[0009] Alternatively, in the recognition step, a line in the camera image may be recognized when the entire anchor passes through or passes close to the line. Thus, an anchor-based method can be used to recognize the line. The entire anchor means an anchor that is not limited to the base anchor point, for example, a line-shaped anchor. "Close" means, for example, a deviation of a few pixels.

[0010] The techniques presented herein further provide an apparatus configured to execute, control, or implement the steps of the method variations presented herein in a corresponding device. This variation of the techniques in the form of an apparatus allows for a rapid and efficient solution to the problem underlying the techniques.

[0011] For this purpose, the device may include at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface for a sensor or actuator for reading a sensor signal from a sensor or outputting a data or control signal to an actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The computing unit may be, for example, a signal processor or a microcontroller, and the memory unit may be a flash memory, an EEPROM, or a magnetic memory unit. The communication interface may be configured to read or output data wirelessly and / or via a wired connection, and a communication interface that can read or output data via a wired connection may, for example, read or output this data electrically or optically from or to a corresponding data transmission line.

[0012] In this specification, a device may refer to an electrical device that processes sensor signals and outputs control and / or data signals accordingly. The device may have an interface that can be configured as hardware and / or software. In a hardware configuration, the interface may be part of a so-called system ASIC that includes various device functions. However, it is also possible for the interface to be a separate integrated circuit or to consist at least in part of discrete elements. In a software configuration, the interface may be a software module that resides, for example, on a microcontroller together with other software modules.

[0013] Also advantageous is a computer program product or computer program which can be stored on a machine-readable carrier or storage medium, such as a semiconductor memory, a hard disk memory or an optical memory, and which comprises a program code which is used to run, implement and / or control the steps of the method according to one of the above-described embodiments, in particular when the program product or program is run on a computer or device.

[0014] A recognition system for a machine or transmission system comprises the above-mentioned device and a camera for providing a camera signal. Such a recognition system can function as a comprehensive system for automatically recognizing lines such as lanes, and the device can predict lines such as lanes particularly uniformly. The recognition system can also be part of a driver assistance system for a vehicle.

[0015] Exemplary embodiments of the techniques presented herein are illustrated in the drawings and explained in more detail in the following description. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic diagram of a machine including an apparatus for recognizing a line in a camera image according to an exemplary embodiment; [Figure 2] 10 is a schematic diagram of a linear regression generated using a generating device of an apparatus in accordance with an exemplary embodiment; [Figure 3] 1 is a schematic illustration of a camera image used by an apparatus in accordance with an exemplary embodiment; [Figure 4] 1 is a flow diagram of a method according to an example embodiment for recognizing lines in a camera image for a machine or transmission system. DETAILED DESCRIPTION OF THE INVENTION

[0017] In the following description of preferred exemplary embodiments of the present technique, elements that are shown in the various figures and that function similarly will be designated by the same or similar reference numerals and will not be described repeatedly.

[0018] FIG. 1 shows a schematic diagram of a machine 100 comprising an apparatus 105 according to an example embodiment for recognizing a line 110 in a camera image. By way of example only, the device 105 according to this exemplary embodiment is arranged on or in the machine 100, here exemplarily designed as a vehicle. According to an exemplary embodiment, the device 105 is implemented in a control device of the machine 100. According to this exemplary embodiment, the machine 100 further comprises a camera 120, here exemplarily designed as an environment capture camera for capturing the environment of the machine 100. According to an alternative exemplary embodiment, the device 105 can be used with a corresponding transmission system.

[0019] The apparatus 105 includes a reading interface 125, a recognition device 130, and a generation device 135. The reading interface 125 is configured to read a camera signal 140 from an interface for the camera 120, the camera signal 140 representing a camera image of the camera 120. The recognition device 130 is configured to recognize the presence of a line 110 in the camera image using an anchor 145 having at least one base anchor point 150. The generation device 135 is configured to generate a linear regression 155 of the line 110 using the at least one base anchor point 150, the linear regression 155 having a first regression point shifted relative to the base anchor point to recognize the line 110, a second regression point generated relative to the first regression point, and at least one further regression point generated relative to a previously generated regression point, similar to the second regression point. A more detailed representation of the linear regression 155 is shown in FIG. 2.

[0020] According to this exemplary embodiment, the camera 120 is designed as a vehicle camera, for example, a vehicle camera of a vehicle's environment capture system. The anchor 145 comprises a template or pattern that is searched for in the camera image by the recognition device 130 to recognize the line 110. According to this exemplary embodiment, the generation device 135 is configured to generate a linear regression 155 that corrects for displacements (also called "offsets"). The apparatus 105 presented herein enables the linear regression 155 (which can also be called a "regression representation") in lane recognition, whereby the offset / deviation of each point along the line 110, which according to this exemplary embodiment is a lane, is predicted not with reference to the anchor 145 but with reference to the previous point. This advantageously results in a particularly uniform prediction of the line 110 (see also FIG. 2).

[0021] Optionally, the apparatus 105 according to this exemplary embodiment further comprises an output device 160, which is configured to output a control signal 165 for controlling a function 170 of the machine 100, here a vehicle, or alternatively a transmission system, using the linear regression 155. Thus, the linear regression 155, which can predict the course of the line 110, e.g., in the form of a lane, can be used to control the machine 100, e.g., to keep the vehicle within the lane.

[0022] According to this exemplary embodiment, the recognition device 130 is configured to use anchors 145 to recognize lines 110 within edge regions of a camera image and / or within cells of a camera image divided into multiple cells (see also FIG. 3).

[0023] According to this exemplary embodiment, the recognition device 130 is configured to recognize the line 110 in the camera image when at least one base anchor point 150 of the anchor 145 passes through the line 110. Alternatively, according to an exemplary embodiment, the recognition device 130 is configured to recognize the line 110 in the camera image when only the base anchor point 150 passes through or passes near the line 110. Thus, according to an exemplary embodiment, an anchor-less method is used to recognize the line 110, and only the base anchor point 150 is required as the anchor 145. According to an alternative exemplary embodiment, the recognition device 130 is configured to recognize the line 110 in the camera image when the entire anchor passes through or passes near the line 110. Thus, according to another exemplary embodiment, an anchor-based method is used to recognize the line 110. The entire anchor 145 refers to an anchor 145 that is more than just the base anchor point 150, for example, a line-shaped anchor. "Near" means, for example, a deviation of up to 5 mm.

[0024] Together with the camera 120, the device 105 presented herein may also be referred to as a perception system 175. According to an exemplary embodiment, the perception system 175 is part of a driver assistance system 180 of the vehicle.

[0025] The device 105 presented in this application enables delta regression / feedback for lane recognition. Traditionally, deep object detectors used in computer vision are based on a concept called "anchors," where so-called "prior boxes" are used as prototypes for recognition. In this context, the detector predicts the shift / offset of a box relative to the respective anchor. Lane recognition is a special case of object recognition where objects are characterized by lines rather than by boxes. Therefore, in addition to traditional instance segmentation methods, anchor-based object recognition methods are also used.

[0026] The device 105 presented herein modifies the regression representation of the lane recognition task by predicting the offset / deviation of each point along the lane with respect to the previous point, rather than with respect to the anchor 145. The approach presented herein can be formulated as both an anchor-based method and an anchor-less method. In both cases, only the base anchor points 150 (cell centers) from the anchor 145 are used, and the difference is only in the fitting step.

[0027] This formulation allows for a more uniform prediction of line 110 compared to a similarly possible regression representation where all regression points are generated relative to an anchor. This also results in a smaller neural network, as anchor 145 is no longer required.

[0028] The device 105 presented herein operates using the following types of data, obtained by receiving digital images and / or videos according to an exemplary embodiment, to calculate control signals 165 for controlling physical systems, such as:

[0029] a computer-controlled machine 100, such as a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system; or - Systems for transmitting information, such as surveillance systems or medical, e.g. imaging systems.

[0030] The device 105 does this by recognizing the presence of objects in the sensor data, particularly lanes or other types of line-based road markings. The device 105 operates with images, and therefore there must be image capture in the form of camera images to be used as input for the techniques presented here.

[0031] 2 shows a schematic diagram of a linear regression 155 generated using a generating device of an apparatus according to an exemplary embodiment, which may be the apparatus described in FIG. The linear regression 155 has a first regression point 200 that is shifted relative to the anchor base anchor point 150. Starting from the first regression point 200, further regression points 210 are generated relative to the previously generated regression points 200, 210, respectively. For example, the generation device according to this exemplary embodiment is configured to generate the linear regression 155, and to recognize the line 110, the first regression point 200 is shifted on the x-axis relative to the base anchor point 150, the first further regression point 210 is generated shifted on the x-axis and y-axis relative to the previously generated first regression point 200, and the subsequent further regression points 210 are generated shifted on the x-axis and y-axis relative to the previously generated further regression points 210, respectively. According to this exemplary embodiment, the linear regression 155 has at least five further regression points 210, each generated by shifting the regression points on the x-axis and y-axis relative to the previously generated regression points 200, 210.

[0032] In known regressions, coordinates are predicted relative to anchors. That is, the prediction represents the displacement in image coordinates that must be applied to the anchors to fit the line to the image only on the x-axis. Unlike known regression methods, the approach presented herein is implemented so that not only the x-coordinate is predicted, but also the y-coordinate is not fixed / fixed. That is, the approach presented herein differs from known regression methods in that the reference for regression is not the line but the points 150, 200, 210. For each cell in which a line is recognized on the image (see FIG. 3), the regression coordinate is predicted relative to the previous coordinate. That is, for the first point 200 on the line, the prediction is made relative to the base anchor point 150, for the first further point 210, the prediction is made relative to the first point 200, and so on. Unlike known methods, this regression is formulated by predicting not only the x-coordinate but also the y-coordinate.

[0033] The approach described here can be formulated in two ways depending on how the classification target is selected. On the one hand, an anchorless method is possible in which only the base anchor points 150 are used for both the linear regression 155 and the classification, i.e., the line 110 in the form of a ground truth line coincides with the anchor when it passes through or near the base anchor point 150, and there is no "coincidence" between the anchor line and the ground truth line.

[0034] Alternatively, an anchor-based method is also feasible, where only the base anchor points 150 are used for the linear regression 155, but the entire anchor is used for classification, not just the base anchor points 150. That is, the line 110 in the form of a ground truth line is matched with the anchor when the similarity between the anchor and the line 110 is high enough.

[0035] Unlike regression predictions, which describe the shift of a line shape relative to its anchors, and where applying an offset to the anchor coordinates results in a line prediction, the present approach changes this in that the anchors are simply points 150, not lines, and the prediction describes the shift relative to the previous point, not the corresponding anchor point. Thus, the linear regression 155 does not represent the line as a whole.

[0036] 3 shows a schematic diagram of a camera image 300 for use in an apparatus according to an exemplary embodiment, which may be the apparatus described in FIG. According to this exemplary embodiment, the recognition device of the apparatus is configured to overlay a grid 305 on the camera image 300 and recognize the presence of the line 110 when the line 110 is recognized in at least one cell 310, 311, 312 of the grid 305. According to this exemplary embodiment, the recognition device is configured to perform classification of the recognized object to recognize the line 110, for example, using a feature extractor. The line 110 represents a road marking according to this exemplary embodiment, or an edge of an object according to an alternative exemplary embodiment. For example, the line 110 according to this exemplary embodiment represents a portion of a lane. For example, the generation device according to this exemplary embodiment is configured to generate a linear regression when base anchor points are recognized at cell centers 315 of cells 310, 311, 312 of the grid 305 on the line 110.

[0037] In summary, according to an exemplary embodiment, additional anchors are defined in cells 310, 311, 312 along the boundary of image 300 corresponding to anchors 145 in cell 310 or corresponding only to the base anchor points of anchors 145 in cell 310, and the classification output predicts, for each cell 310, 311, 312, whether the cell 310, 311, 312 contains a corresponding line 110, e.g., a lane, that is similar to one of the anchors 145 in that cell 310, 311, 312. For clarity, only one corresponding anchor 145 passing through one of the cells 310 is shown in FIG. 3 .

[0038] In other words, according to an exemplary embodiment, the network predicts, for each cell 310, 311, 312, whether a line 110 similar to the anchor 145 assigned to the respective cell 310, 311, 312 exists within the cell 310, 311, 312. According to this exemplary embodiment, the grid 305 includes eight uniformly sized cells 310, 311, 312 arranged side by side along two opposite image side edges of the camera image 300. According to this exemplary embodiment, the grid 305 has twelve uniformly sized cells 310, 311, 312 arranged adjacently along the bottom edge of the camera image 300, where cell 310 represents a positive prediction of the line 110, cell 311 represents a negative prediction, and cell 312 represents a boundary case.

[0039] 4 shows a flow chart of a method 400 according to an example embodiment for recognizing lines in a camera image for a machine or transmission system, which may be a method 400 executable by one of the devices described with reference to the preceding figures.

[0040] Method 400 includes a reading step 405, a recognition step 410, and a generation step 415. In the reading step 405, a camera signal is read from an interface for a camera, the camera signal representing a camera image of the camera. In the recognition step 410, the presence of a line in the camera image is recognized using an anchor having at least one base anchor point. In the generation step 415, a linear regression of the line is generated using the at least one base anchor point, the linear regression having a first regression point shifted relative to the base anchor point and at least one further regression point generated relative to a previously generated regression point to recognize the line.

[0041] If multiple anchors exist, they can be used to identify a line or to identify different lines in a corresponding way. Optionally, the method 400 according to this exemplary embodiment further comprises an outputting step 420. In the outputting step 420, a control signal is output using the linear regression for controlling a function of a machine, in particular a vehicle, or a transmission system.

Claims

1. A method (400) for recognizing a line (110) in a camera image (300) for a machine (100) or a transmission system, comprising: a step (405) of reading a camera signal (140) from an interface for a camera (120), the camera signal (140) representing the camera image (300) of the camera (120); Recognizing (410) the presence of said line (110) in said camera image (300) using at least one anchor (145) having at least one base anchor point (150); generating (415) a linear regression (155) of the line (110) using the at least one base anchor point (150), the linear regression (155) having a first regression point (200) shifted relative to the base anchor point (150) to recognize the line (110) and at least one further regression point (210) generated relative to a previously generated regression point; A method (400) comprising:

2. 2. The method (400) of claim 1, comprising an output step (420), in which the linear regression (155) is used to output a control signal (165) for controlling a function (170) of the machine (100), in particular a vehicle, or the transmission system.

3. 3. A method (400) according to claim 1 or 2, wherein in the recognition step (410), the line (110) is recognized within an edge region of the camera image (300) using the at least one anchor (145) and / or within a cell (310) of the camera image (300) divided into a plurality of cells (310, 311, 312) using the anchor (145) of each of the cells (310).

4. 4. The method (400) of claim 1, wherein in the recognition step (410), the line (110) in the camera image (300) is recognized when the at least one base anchor point (150) of the anchor (145) passes through the line (110).

5. 5. The method (400) of claim 4, wherein in the recognizing step (410), the line (110) in the camera image (300) is recognized when the entire anchor passes through the line (110).

6. The method (400) of any one of claims 1 to 5, wherein the line (110) represents a road marking or an edge of an object.

7. An apparatus (105) designed to perform and / or control the steps (405, 410, 415, 420) of the method (400) according to any one of claims 1 to 6 in a corresponding unit (125, 130, 135, 160).

8. A recognition system (175) for a machine (100) or a transmission system, comprising the device (105) of claim 7 and a camera (120) for providing said camera signal (140).

9. A computer program designed to perform and / or control the steps (405, 410, 415, 420) of the method (400) according to any one of claims 1 to 6.

10. A machine-readable storage medium storing the computer program according to claim 9.

Images