Method and device for identifying lines in camera image for machine or transmission system, and identification system for machine or transmission system

Abstract

The proposed solution relates to a method for a machine (100) for identifying lines (110) in a camera image. The method has a reading step, an identifying step and a generating step. In a reading step, a camera signal (140) is read from an interface to a camera (120), the camera signal (140) representing a camera image of the camera (120). In an identification step, the presence of the line (110) in the camera image is identified using an anchor point (145) having at least one base anchor point (150). In the generating step, a linear regression (155) of the line (110) is generated using at least a base anchor point (150), the linear regression (155) having a first regression point displaced relative to the base anchor point (150) and at least one further regression point generated relative to the previously generated regression point in order to identify the line (110).

Description

Technical Field

[0001] The invention relates to a method or an apparatus of the type specified in the independent claims. The subject matter of the invention is also a computer program. Background Art

[0002] Driver assistance systems use lane recognition functions, for example, for lane assistance. Summary of the invention

[0003] Against this background, the solution proposed here provides, according to the main claim, a method for detecting lines in a camera image for a machine or a transport system, a device for applying the method, a corresponding computer program and finally a detection system for a machine or a transport system. The measures listed in the dependent claims can advantageously expand and improve the method specified in the independent claim.

[0004] The advantage achieved by this approach is that a particularly uniform prediction of the lines can be achieved.

[0005] A method for identifying a line in a camera image for a machine or a transport system is proposed. The method comprises a reading step, a recognition step and a generation step. In the reading step, a camera signal is read from an interface to a camera, wherein the camera signal represents a camera image of the camera. In the recognition step, the presence of a line in the camera image is recognized using at least one anchor point, according to one embodiment, using a plurality of anchor points each having at least one base anchor point. In the generation step, a linear regression of the line is generated using at least the base anchor point, wherein the linear regression has a first regression point displaced relative to the base anchor point and at least one further regression point generated relative to a previously generated regression point, in order to identify the line.

[0006] The method can be implemented, for example, in software or hardware or in a hybrid form of software and hardware, for example in a control unit.

[0007] The machine may be a fully or partially computer-controlled machine, such as a vehicle. The transmission system may be a system for transmitting information, such as a monitoring system or a medical system, such as an imaging system. The camera may be a vehicle camera, such as a vehicle camera of an environmental detection system of a vehicle. In the recognition step, a grid may be placed on the camera image, for example, and the presence of a line may be recognized if a line is recognized in at least one cell of the grid. In the recognition step, for example, a classification of the recognized objects may be performed, for example, using a feature extractor, in order to recognize the line. The line may represent a lane marking or an edge of an object. The line may represent, for example, a portion of a lane. The anchor point or the anchor points may have a sample or pattern, which is searched for in the camera image. For example, different anchor points may be used to recognize different lines in the camera image. In the generation step, a linear regression may be generated, which compensates for an offset, also called an "offset". For example, if the base anchor point is recognized in the cell center of a cell of the grid and / or on a line, a linear regression may be generated in the generation step. For example, a linear regression can be generated in a generation step, wherein a first linear regression point is shifted relative to a base anchor point on the X axis and a first further regression point is generated shifted relative to the first regression point on the X axis and on a Y axis extending perpendicular to the X axis, and the following further regression points are generated shifted relative to the respectively previously generated regression point on the X axis and on the Y axis, in order to identify the line. This method enables linear regression in lane recognition in that it predicts the offset / deviation of each point along the line / lane relative to the previous point and not to the anchor point, and the linear regression can also be referred to as a "regression representation". This advantageously leads to a particularly uniform prediction of the line. The prediction describes the offset in the image coordinates, which must be applied to the anchor point in order to fit the line into the image.

[0008] The method further comprises an output step in which a control signal is output for controlling a function of a machine, in particular a vehicle, or a transmission system using a linear regression. Thus, a linear regression which is able to predict the course of a line, for example in the form of a lane, can be used to control a vehicle, for example to keep the vehicle in the lane.

[0009] According to one embodiment, in the recognition step, a line can be recognized in an edge region of a camera image using at least one anchor point and / or in a cell of a camera image divided into a plurality of cells using the corresponding anchor point of the cell. For example, in the recognition step, a line can be recognized in the cell center of a cell. In this way, a line can be searched for quickly and easily in each cell in the edge region of a camera image.

[0010] Furthermore, it is advantageous that, according to one embodiment, in the recognition step, a line is recognized in the camera image when at least a base anchor point of the anchor points extends through the line. However, in the recognition step, a line can also be recognized in the camera image when only the base anchor point extends through the line or extends near the line. Thus, an anchor-free method can be used to recognize the line, in which only the base anchor point is required as an anchor point.

[0011] Alternatively, in the recognition step, a line can be recognized in the camera image when the overall anchor point passes through the line or extends near the line. Thus, the line can be recognized using an anchor point-based method. An overall anchor point is to be understood as an anchor point that has more than one basic anchor point, such as a linear anchor point. "Nearby" is to be understood as a deviation of a few pixels, for example.

[0012] The solution proposed here also provides a device, which is configured to perform, control or implement the steps of the variant of the method proposed here in a corresponding facility. The implementation variant of the solution in the form of a device can quickly and effectively solve the task on which the solution is based.

[0013] For this purpose, the device can have at least one computing unit for processing signals or data, at least one storage unit for storing signals or data, at least one interface to a sensor or actuator for reading sensor signals from a sensor or for outputting data signals or control signals to an actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The computing unit can be, for example, a signal processor, a microcontroller or the like, wherein the storage unit can be a flash memory, an EEPROM or a magnetic memory. The communication interface can be configured to read or output data wirelessly and / or by wire connection, wherein a communication interface that can read or output data by wire connection can, for example, read data from a corresponding data transmission line or output data to a corresponding data transmission line electrically or optically.

[0014] The device can be understood here as an electronic device that processes sensor signals and outputs control signals and / or data signals accordingly. The device can have an interface, which can be constructed in hardware and / or in software. When constructed in hardware, the interface can be, for example, a part of a so-called system ASIC, which contains different functions of the device. However, it can also be that the interface is its own integrated circuit or is at least partially composed of discrete components. When constructed in software, the interface can be a software module, which, for example, exists on a microcontroller in parallel with other software modules.

[0015] Also advantageous is a computer program product or a computer program with a program code, which can be stored on a machine-readable carrier or storage medium, such as a semiconductor memory, a hard disk memory or an optical memory, and is used to execute, implement and / or control the steps of a method according to one of the above-described embodiments, in particular when the program product or the program is executed on a computer or a device.

[0016] A recognition system for a machine or a transport system has the aforementioned device and a camera for providing a camera signal. Such a recognition system can be used as a comprehensive system for automatically recognizing lines, such as lanes, wherein, due to the device, lines, such as lanes, can be predicted particularly uniformly. The recognition system can be part of a driver assistance system for a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] An exemplary embodiment of the solution proposed here is shown in the drawings and is further explained in the following description. The drawings show:

[0018] Figure 1 : A schematic diagram of a machine according to an embodiment, the machine having a device for identifying lines in a camera image;

[0019] Figure 2 : A schematic diagram of a linear regression generated using a generation facility of a device according to an embodiment;

[0020] Figure 3 : A schematic diagram of a camera image for use with an apparatus according to an embodiment;

[0021] Figure 4 : Flowchart of a method according to one embodiment for identifying lines in a camera image for a machine or a transport system.

[0022] In the following description of advantageous exemplary embodiments of the present solution, identical or similar reference symbols are used for similarly acting elements illustrated in the various figures, wherein a repeated description of these elements is omitted. DETAILED DESCRIPTION

[0023] Figure 1 A schematic diagram of a machine 100 according to an exemplary embodiment is shown, which has a device 105 for detecting a line 110 in a camera image.

[0024] Merely by way of example, according to this embodiment, the device 105 is arranged on or in the machine 100, which is exemplarily configured as a vehicle here. According to one embodiment, the device 105 is implemented in a controller of the machine 100. According to this embodiment, the machine 100 also has a camera 120, which is exemplarily configured as an environment detection camera here, for detecting the surrounding environment of the machine 100. According to an optional embodiment, the device 105 can be used together with a transmission system accordingly.

[0025] The device 105 has a reading interface 125, a recognition facility 130 and a generation facility 135. The reading interface 125 is designed to read a camera signal 140 from an interface connected to the camera 120, wherein the camera signal 140 represents a camera image of the camera 120. The recognition facility 130 is designed to recognize the presence of the line 110 in the camera image using an anchor point 145 having at least one base anchor point 150. The generation facility 135 is designed to generate a linear regression 155 of the line 110 using at least the base anchor point 150, wherein the linear regression 155 has a first regression point shifted relative to the base anchor point and a second regression point generated relative to the first regression point and at least one further regression point correspondingly relative to the second regression point, the at least one further regression point being generated relative to the previously generated regression point in order to recognize the line 110. Figure 2 A more precise description of linear regression 155 is shown in FIG.

[0026] According to this embodiment, the camera 120 is configured as a vehicle camera, for example a vehicle camera of a surrounding detection system of a vehicle. The anchor point 145 comprises a sample or a pattern, which is searched for in the camera image by means of the recognition facility 130 in order to recognize the line 110. According to this embodiment, the generation facility 135 is configured for generating a linear regression 155 which compensates for an offset also referred to as a bias. The device 105 proposed here enables the implementation of a linear regression 155, which can also be referred to as a “regression representation”, in lane recognition by predicting the offset / deviation of each point along the line 110, which according to this embodiment is a lane, relative to the preceding point, rather than relative to the anchor point 145. This advantageously leads to a particularly uniform prediction of the line 110, which can also be seen Figure 2 .

[0027] Optionally, according to this exemplary embodiment, the device 105 also has an output device 160, which is designed to output a control signal 165 for controlling the machine 100, here a vehicle, or alternatively a function 170 of a transport system, using the linear regression 155. Thus, the machine 100 can be controlled using the linear regression 155, which can predict the course of the line 110, for example in the form of a lane, for example to keep the vehicle within the lane.

[0028] According to this exemplary embodiment, the recognition device 130 is designed to recognize the line 110 in the edge region of the camera image and / or in a cell of the camera image divided into a plurality of cells using the anchor point 145, for which purpose it can be seen that Figure 3 .

[0029] According to this embodiment, the recognition device 130 is designed to recognize the line 110 in the camera image when at least the base anchor point 150 of the anchor point 145 extends through the line 110. Alternatively, according to one embodiment, the recognition device 130 is designed to recognize the line 110 in the camera image when only the base anchor point 150 extends through the line 110 or extends in the vicinity of the line 110. Therefore, according to one embodiment, for recognizing the line 110, an anchor-free method is used, in which only the base anchor point 150 is required as the anchor point 145. According to an alternative embodiment, the recognition device 130 is designed to recognize the line 110 in the camera image when the entire anchor point extends through the line 110 or in the vicinity of the line 110. Therefore, according to another embodiment, an anchor-based method is used to recognize the line 110. An anchor point 145 having more than one base anchor point 150 is to be understood as an entire anchor point 145, for example a linear anchor point. Deviations of, for example, up to 5 mm are to be understood as “in the vicinity”.

[0030] The device 105 presented here may also be referred to together with the camera 120 as a recognition system 175. According to one exemplary embodiment, the recognition system 175 is part of a driver assistance system 180 of the vehicle.

[0031] Device 105 proposed here enables a triangular regression / feedback for lane recognition.

[0032] Traditionally, deep object detectors used in computer vision are based on a concept called "anchors", i.e., so-called "prior boxes" are used as prototypes for recognition. In this relationship, the detector predicts the displacement / offset of the box relative to each anchor point. Lane recognition is a special case of object recognition, in which objects are identified not by boxes but by lines. Therefore, in addition to traditional instance segmentation methods, anchor-based object recognition methods are also used.

[0033] The device 105 proposed here changes the regression representation of the lane recognition task in that it predicts the bias / deviation of each point along the lane relative to the previous point, rather than relative to the anchor point 145. The proposed solution can be described not only as an anchor-based method, but also as an anchor-free method. In both cases, only the base anchor point 150 is used as the anchor point 145 (cell center), the difference only being the matching step.

[0034] This representation results in a more uniform prediction of line 110 compared to equally possible regression representations in which all regression points are generated relative to the anchor point. This also results in a smaller neural network because anchor point 145 becomes redundant.

[0035] The device 105 proposed here processes the following types of data, which, according to one embodiment, are obtained by receiving digital images and / or videos, for calculating control signals 165 for controlling a physical system, for example:

[0036] - a computer controlled machine 100, such as a robot, a vehicle, a household appliance, a power tool, a processing machine, a personal assistant or an access control system, or

[0037] - Systems for transmitting information, such as monitoring systems or medical systems, such as imaging systems.

[0038] Device 105 performs this operation by identifying the presence of objects in the sensor data, in particular lanes or other types of line-based road markings.

[0039] The device 105 processes images, so an image record must be available in the form of a camera image, which serves as input for the approach proposed here.

[0040] Figure 2 A schematic diagram of a linear regression 155 is shown, which is generated using a generation facility of a device according to an exemplary embodiment. Figure 1 The device described in .

[0041] The linear regression 155 has a first regression point 200, which is displaced relative to the anchor point 150 of the anchor point. Starting from the first regression point 200, further regression points 210 are generated, which are each generated relative to the immediately preceding regression point 200, 210. For example, the generating device is designed according to this embodiment to generate a linear regression 155, wherein the first regression point 200 is displaced relative to the anchor point 150 on the X axis, the first further regression point 210 is generated displaced relative to the preceding first regression point 200 on the X axis and the Y axis, and the following further regression points 210 are generated displaced relative to the preceding further regression points 210 on the X axis and the Y axis, in order to identify the line 110. According to this embodiment, the linear regression 155 has at least five further regression points 210, each of which is generated displaced relative to the preceding regression point 200, 210 on the X axis and the Y axis.

[0042] In known regressions, coordinates are predicted relative to an anchor point, i.e. the predictions describe an offset in image coordinates which must be applied to the anchor point in order to fit the line into the image only on the X axis. Unlike known regression methods, the proposed solution is implemented in such a way that not only the X coordinates are predicted, but also the Y coordinates are not fixed / stationary. The proposed solution thus differs from known regression methods in that the reference for the regression is not a line, but a point 150, 200, 210, and for each cell through which a line has been identified on the image, see Figure 3 , the regression coordinates are predicted relative to the previous coordinates. This means that for the first point 200 in the line a prediction is made relative to the base anchor point 150, for the first further point 210 a prediction is made relative to the first point 200, and so on. This regression is formulated in contrast to the known solutions in that not only the X coordinate but also the Y coordinate is predicted.

[0043] The scheme described here can be formulated in two ways, depending on how the classification target is chosen.

[0044] On the one hand, an anchor-free method can be implemented in which only the base anchor point 150 is used not only for linear regression 155 but also for classification. This means that when a line 110 in the form of a ground truth line passes through the base anchor point 150 of an anchor point or extends near the base anchor point, the line is consistent with the anchor point, and there is no anchor point line "consistent" with the ground truth line, or

[0045] On the other hand, an anchor-based approach may be implemented in which only base anchors 150 are used for linear regression 155, while an overall anchor with more than one base anchor 150 is used for classification. This means that if the similarity between an anchor and a line 110 in the form of a ground truth line is still high enough, the line 110 is aligned with the anchor.

[0046] Unlike the regression where the prediction accounts for linear displacements relative to a linear anchor point, where applying a bias on the anchor point coordinates results in a linear prediction, the present solution changes this situation as follows: the anchor point is just a point 150 instead of a line, and the prediction accounts for displacements relative to the preceding point, not relative to the corresponding anchor point. Therefore, the linear regression 155 does not represent a straight line as a whole.

[0047] Figure 3 A schematic diagram of a camera image 300 is shown for use with a device according to an exemplary embodiment. Figure 1 Or the device described in 2.

[0048] According to this embodiment, the recognition facility of the device is designed to place a grid 305 on the camera image 300 and to recognize the presence of the line 110 if the line 110 is recognized in at least one cell 310, 311, 312 of the grid 305. According to this embodiment, the recognition facility is designed to perform a classification of the recognized object, for example using a feature extractor, to recognize the line 110. According to this embodiment, the line 110 represents a lane marking or, according to an alternative embodiment, represents the edge of an object. For example, according to this embodiment, the line 110 represents a part of a lane. For example, according to this embodiment, the generation facility is designed to generate a linear regression if a base anchor point is recognized on the line 110 in the cell center 315 of the cell 310, 311, 312 of the grid 305.

[0049] In summary, according to one embodiment, corresponding to the anchor point 145 in the cell 310 or only the base anchor point corresponding to the anchor point 145 in the cell 310, additional anchor points are defined in the cells 310, 311, 312 along the boundary of the image 300, and the classification output predicts for each cell 310, 311, 312 whether the cell 310, 311, 312 contains a corresponding line 110, such as a lane, which approximates one of the anchor points 145 on the cell 310, 311, 312. Figure 3 For clarity, only one corresponding anchor point 145 passing through one of the cells 310 is drawn.

[0050] In other words, according to one exemplary embodiment, the network predicts for each cell 310, 311, 312 whether a line 110 is present in the cell 310, 311, 312, which line approximates the anchor point 145 assigned to the corresponding cell 310, 311, 312. According to this exemplary embodiment, the grid 305 comprises eight uniformly sized cells 310, 311, 312 arranged one above the other along the edges of two opposite image sides of the camera image 300. Along the lower edge of the camera image 300, the grid 305 has twelve uniformly sized cells 310, 311, 312 arranged adjacent to each other according to this exemplary embodiment. Cell 310 represents a positive prediction for the line 110, cell 311 represents a negative prediction, and cell 312 represents a borderline case.

[0051] Figure 4 A flow chart of a method 400 for detecting lines in a camera image for a machine or a transport system according to an exemplary embodiment is shown. The method 400 may be a method that can be implemented by one of the devices described with reference to the preceding figures.

[0052] The method 400 has a reading step 405, an identification step 410, and a generating step 415. In the reading step 405, a camera signal is read from an interface to a camera, wherein the camera signal represents a camera image of the camera. In the identification step 410, the presence of a line in the camera image is identified using anchor points having at least one base anchor point. In the generating step 415, a linear regression of the line is generated using at least the base anchor point, wherein the linear regression has a first regression point displaced relative to the base anchor point and at least one further regression point generated relative to a previously generated regression point, to identify the line.

[0053] If multiple anchor points are present, these can be used in a corresponding manner to identify the line or to identify a different line.

[0054] Optionally, the method 400 according to this exemplary embodiment further comprises an output step 420. In the output step 420, a control signal is outputted for controlling a function of a machine, in particular a vehicle, or a transmission system, using linear regression.

Claims

1. A method (400) for detecting a line (110) in a camera image (300) for a machine (100) or a transport system, wherein: The method (400) comprises the following steps: reading (405) a camera signal (140) from an interface connected to a camera (120), wherein the camera signal (140) represents the camera image (300) of the camera (120); identifying (410) the presence of the line (110) in the camera image (300) using at least one anchor point (145) having at least one base anchor point (150); and A linear regression (155) of the line (110) is generated (415) using at least the base anchor point (150), wherein the linear regression (155) has a first regression point (200) and at least one additional regression point (210), the first regression point being displaced relative to the base anchor point (150), and the at least one additional regression point being generated relative to a previously generated regression point to identify the line (110).

2. The method (400) according to the preceding claim comprises an output step (420), in which a control signal (165) is output using the linear regression (155) to control a function (170) of the machine (100), in particular a vehicle or the transmission system.

3. A method (400) according to any one of the preceding claims, in which, in the identification step (410), the line (110) is identified in an edge area of ​​the camera image (300) using the at least one anchor point (145) and / or in a cell (310) of the camera image (300) divided into a plurality of cells (310, 311, 312) using the corresponding anchor point (145) of the cell (310).

4. A method (400) according to any of the preceding claims, in which, in an identification step (410), the line (110) is identified in the camera image (300) when at least the base anchor point (150) of the anchor point (145) extends through the line (110).

5. The method (400) of claim 4, wherein in the identifying step (410), the line (110) is identified in the camera image (300) when an overall anchor point extends through the line (110).

6. The method (400) according to any one of the preceding claims, wherein: The lines (110) represent lane markings or edges of objects.

7. A device (105) which is configured to implement and / or control the steps (405, 410, 415, 420) of the method (400) according to any of the preceding claims in a corresponding unit (125, 130, 135, 160).

8. An identification system (175) for a machine (100) or a transport system, wherein: The recognition system (175) has a device (105) according to claim 7 and a camera (120) for providing the camera signal (140).

9. A computer program, which is configured to implement and / or control the steps (405, 410, 415, 420) of the method (400) according to any one of claims 1 to 6.

10. A machine-readable storage medium having stored thereon the computer program according to claim 9.

Images