Method, device and computer program for detecting an object

By using time filter scheme and module series connection in the detection system, the inaccuracy problem of object detection in poor environments is solved, and reliable detection and calculation efficiency improvement in nighttime, rain and snow conditions are achieved.

CN109747641BActive Publication Date: 2024-11-26ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201811307042.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-11-06
Filing Date
2018-11-05
Publication Date
2024-11-26
Estimated Expiration
2038-11-05

AI Technical Summary

Technical Problem

The prior art is difficult to accurately detect objects under poor environmental conditions (such as nighttime, rain and snow, rapid brightness changes, tunnel driving, etc.), and is sensitive to inaccurate environmental records, which affects the detection effect.

Method used

Using a time filter scheme, filtering and decoupling module output parameters through series-connected filter modules and memory modules, training system using sparse markers, including standardization, activation and LSTM modules, using jumper connection and decoding modules To improve system robustness and computing efficiency.

Benefits of technology

Reliable detection of objects is achieved under poor environmental conditions, improving the robustness and computing efficiency of the system, being able to process incompletely labeled training data, and enhancing tolerance for inaccurate environmental records.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN109747641B_ABST
    Figure CN109747641B_ABST
Patent Text Reader

Abstract

A method for detecting an object by means of a system, an apparatus for implementing the method, a computer program and a machine-readable storage element storing the method, the method comprising: providing an input parameter of a first module of modules connected in series by means of a detection unit, determining the output parameter of each module step by step along the modules connected in series starting from the first module according to the corresponding module input parameter and parameterization, a filter module filtering the output parameter of a predeterminable module and determining the output parameter according to the output parameter and the temporally preceding intermediate parameter, the filter module determining the temporally latest intermediate parameter according to the temporally preceding output parameter of the predeterminable module and the temporally preceding intermediate parameter determined by the filter module according to the temporally preceding output parameter of the predeterminable module, the filter module output parameter and at least one input parameter of one of the preceding modules of the predeterminable module are combined into a memory parameter by means of a memory module and provided as an input parameter of the next module.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for detecting an object by means of a system. The system comprises a plurality of modules connected in series one after the other, wherein at least one connection between a predeterminable module and its immediately following module is respectively made via a filter module and a memory module connected in series. The invention also relates to a device and a computer program which are designed to implement the method. Background Art

[0002] A method for detecting the presence of a vehicle in a traffic situation using a neural network is known from US Pat. No. 544,848 A. The neural network comprises a plurality of input units, a plurality of hidden units and a plurality of output units, wherein the output of each of the plurality of input units is connected to the input of each of the plurality of hidden units. The output is connected from each of the plurality of hidden units to the input of each of the plurality of output units. The neural network generates an output signal at each of the plurality of output units and indicates whether a vehicle has been detected in the detection zone. Summary of the invention

[0003] Advantages of the present invention

[0004] In contrast, the method of the present invention has the advantage that the temporal filtering of the input parameters is performed by a filter scheme, thereby achieving robustness relative to inaccurate environmental recordings, especially inaccurate environmental recordings of traffic scenes. Therefore, the method itself can be used for detecting objects under poor environmental conditions. Poor environmental conditions may lead to inaccurate environmental recordings and negatively affect the detection of objects. By means of temporal filtering, inaccuracies and / or ambiguities in the environmental recording can be distinguished, which are formed, for example, due to poor environmental conditions. Thus, the method can also reliably detect the object, for example, at night, in rain / snow conditions, in rapid changes in ambient brightness, for example, in tunnel driving conditions, or in the case where part of the object is temporarily covered. In addition, the calculated parameters can be reused within the system by the special structure of the filter scheme, thereby achieving a parameter-efficient and therefore also computationally efficient system. Likewise, the special structure of the filter concept enables the decoupling of the calculated variables of the individual modules of the system, since the calculated variables are no longer dependent on the output variables of the respective preceding modules of the system, but all output variables of all preceding modules of the system can be used again. Another advantage of the method is that the temporal filtering makes it possible to train the modules even with incompletely labeled training data. The temporal filtering makes it possible to use a large number of sparsely labeled training data for training the system.

[0005] In a first aspect, the invention relates to a method for detecting an object by means of a system. The system comprises a plurality of modules connected in series one after the other. At least one connection between a predeterminable module and its immediately following module is respectively made by means of a filter module and a memory module connected in series. The first of the modules connected in series is connected to a detection unit, and the last of the modules connected in series is connected to a detection module. The method comprises the following steps:

[0006] - providing an input variable for a first module of the modules connected in series by means of a detection unit.

[0007] - Starting with the first module of the modules connected in series, the output variable of each module is determined step by step along the modules connected in series as a function of the input variables of the respective module and the parameterization of the respective module. The output variable of the predeterminable module is filtered by means of a filter module, wherein the filter module determines the output variable as a function of the output variable of the predeterminable module and a temporally preceding intermediate variable. The filter module determines the temporally latest intermediate variable as a function of the temporally preceding output variable of the predeterminable module and the temporally preceding intermediate variable. The temporally preceding intermediate variable has been determined as a function of the temporally preceding output variable of the predeterminable module by means of the filter module. The output variable of the filter module and at least one input variable of one of the preceding modules of the predeterminable module are combined by means of a memory module into a memory variable and provided as an input variable for the immediately following module.

[0008] Step-by-step determination can be understood as: starting with the first module, determining its output parameter and then, when the output parameter of the previous module has been determined, the next module determines the output parameter based on the output parameter of the module immediately preceding it, and more precisely, determining step by step until the last module has determined its output parameter.

[0009] A temporally preceding intermediate variable may be understood as an intermediate variable from a temporal sequence of intermediate variables, which is assigned to a predeterminable, preceding, in particular, immediately preceding, point in time. A temporally latest intermediate variable may be understood as an intermediate variable from a temporal sequence of intermediate variables, which is assigned to a predeterminable, latest point in time. A temporally preceding output variable of a predeterminable module may be understood as an output variable from a temporal sequence of output variables, which is assigned to a predeterminable, preceding, in particular, immediately preceding, point in time.

[0010] In a particularly advantageous embodiment of the first aspect of the invention, the filter module filters the output variable of the predefinable module as a function of a temporally preceding intermediate variable, wherein the temporally preceding intermediate variable characterizes the temporal development of the output variable of the predefinable module.

[0011] In another advantageous embodiment, the filter module comprises a normalization module and an activation module and an LSTM module (Long Short Term Memory-Module). These modules each comprise a machine learning system. The temporally latest intermediate variable is determined by means of the LSTM module from the input variables of the LSTM module and the temporally preceding intermediate variables.

[0012] The machine learning system may be, for example, a deep neural network. In this context, a deep neural network may be understood in particular to mean a neural network having a plurality of layers.

[0013] A normalization module can be understood as a module that determines its output variable in such a way that each output variable of the normalization module lies within the same predeterminable digital interval. An activation module can be understood as a module that outputs its input variable at the output in an increased or decreased manner according to a predeterminable multiplication factor.

[0014] It is particularly advantageous if the filter module additionally has a jumper connection (Überbrückungsverbindung) (English: skip connection / shortcut connection). When filtering an output variable of a predefined module by means of the filter module, the input variable of the filter module is forwarded to the output of the filter module by means of the jumper connection. At the output of the filter module, the input variable of the filter module is added to the output variable of the filter module. This has the advantage that the determination of the parameterization of the modules connected in series can be better handled by the jumper connection. This is because the jumper connection can avoid that during the determination of the parameterization, the gradient preferably used for this purpose can become negligibly small or too large. Therefore, a suitable parameterization can be determined more reliably by means of the jumper connection.

[0015] It is also particularly advantageous that the filter module additionally includes a decoding module. The decoding module connects the output ends of the LSTM module and the filter module. When filtering the output parameter of a module that can be predetermined by means of the filter module, the output parameter of the LSTM module is decoded by means of the decoding module so that the output parameter of the filter module has the same number of dimensions as the input parameter of the filter module forwarded by means of the bridging connection. As a result, the number of dimensions of the output parameter of the LSTM module can be selected arbitrarily because the number of dimensions is adapted by means of the decoding module. This has the advantage that information cannot be lost by changing the dimensions by the decoding module and that freedom of choice of the number of dimensions of the output parameter of the LSTM module can be provided.

[0016] Advantageously, the connection between the last module of the series-connected modules and the detection module is effected by means of a third memory module which combines the output variable of the last module with at least one output variable of a filter module connecting two consecutive modules of the series-connected modules to form a memory variable.

[0017] Advantageously, a compression module is added before a predeterminable module, which compresses the input variables of the module, in particular by reducing the dimension of the input variables. This has the advantage that, for example, negligibly small or redundant values ​​that can be included in the output variable of the preceding module can be removed by dimensionality reduction and unnecessary calculations can be saved.

[0018] It is particularly advantageous to detect an object at the output of the last module of the modules connected in series by means of a detector module, wherein the at least semi-autonomous machine is controlled as a function of the detected object. The at least semi-autonomous machine can be, for example, a vehicle, but also a robot.

[0019] In another aspect, the invention relates to a computer program, which is designed to implement one of the aforementioned methods, the computer program therefore comprising instructions which, when the computer program is run on a computer, cause the instructions and the computer to implement one of the aforementioned methods together with all the steps of the method. The invention also relates to a machine-readable memory module on which the computer program is stored, and to an apparatus, which is designed to implement one of the aforementioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Embodiments of the present invention are shown in the attached drawings and are further explained in the following description.

[0021] Figure 1 A schematic diagram of a vehicle is shown, the vehicle including one embodiment of a system for detecting an object;

[0022] Figure 2 A schematic diagram showing an embodiment of a fragment of a system for detecting an object, the system comprising a plurality of modules connected in series one after the other and a filter module and a memory module connected in series;

[0023] Figure 3 A schematic diagram showing an embodiment of a filter module, the filter module comprising a normalization module, an activation module, and an LSTM module;

[0024] Figure 4 A schematic diagram shows an alternative embodiment of a filter module, wherein the filter module additionally has a jumper connection;

[0025] Figure 5 A schematic diagram shows a further alternative embodiment of a filter module, which additionally has a decoding module;

[0026] Figure 6 The schematic diagram shows a specific embodiment of a method which is carried out on a computer for operating a system for detecting an object. DETAILED DESCRIPTION

[0027] Figure 1 An exemplary schematic diagram of a vehicle (10) is shown, which comprises a device (11). The device (11) consists of a plurality of components (12, 13, 14) which are connected in series with one another. The components (12, 13, 14) which are connected with one another can preferably be connected to a computing unit (15). The computing unit (15) can include a memory unit (16).

[0028] The first component is a detection unit (12), which is connected to the system (13) and the component (14) is a detection module. The detection unit (12) is designed to detect the environment of the vehicle (10) and is preferably a camera, but can also have other sensors for environmental detection. The detection module (14) is designed to detect objects based on output variables of the system (13). It is conceivable that the vehicle (10) is controlled based on the detected object, preferably by means of a control device of the vehicle (10), so that, for example, a collision with the object is prevented or the object is avoided.

[0029] Figure 2A schematic diagram of a system (13) for detecting an object according to one embodiment is shown. The system (13) comprises a plurality of modules (210, 211, 212, 213) connected in series one after the other. The individual modules (210, 211, 212, 213) have parameterizations and can preferably each comprise a machine learning system, in particular a deep neural network, which has at least one layer, the at least one layer having neurons. The modules (210, 211, 212, 213) determine output variables of the module based on the respective parameterizations of the module and the respective input variables of the module.

[0030] At least one connection between the modules (210, 211, 212, 213) connected in series, between a predeterminable module (210) and its immediately following module (211), is respectively made with a filter module and a memory module (22, 23) connected in series. The output variable of the predeterminable module (210) is filtered by means of the filter module (22). The filter module (22) determines the output variable of the predeterminable module (210) and the intermediate variable (z t-1 ) to determine the output parameter. The intermediate parameter (z t-1 ) has been determined by the filter module (22) based on the temporally preceding output variable of a predeterminable module (210). The output variable of the filter module (22) is then forwarded to a memory module (23). The memory module (23) combines the determined output variable of the filter module (22) and at least one input variable of one of the preceding modules (210) of the filter module (22) into a memory variable. The input variable of the preceding module (210) of the filter module (22) can be forwarded to the memory module (23) via a connection (201). Optionally, the output variable of the preceding filter module (if any) can also be forwarded to the memory module and taken into account in the above-mentioned memory variable.

[0031] The memory module then stores the combined memory variables. The memory variables of the memory module (23) are subsequently forwarded to the immediately following module (211) of the predefinable module (210).

[0032] Preferably, the filter module (22) uses a temporally preceding intermediate variable (z t-1 ) filters the output variable of the predeterminable module (210). Preferably, in this case, the temporally preceding intermediate variable (z t-1 ) characterizes the temporal development of an output variable of a predeterminable module (210).

[0033] Similarly, in Figure 2 It is shown that further filter modules and memory modules (22, 23) can be connected between further predeterminable modules (212, 213). As soon as the system (13) comprises a plurality of filter modules and memory modules (22, 23), the content of a predeterminable preceding memory module can preferably be forwarded to a subsequent memory module via a connection (202) and taken into account in the memory parameters of the subsequent memory module. This has the advantageous effect that the determined output parameters of the filter module (22) can be aggregated in the memory module (23), forwarded and at least partially reused in the subsequent module.

[0034] Preferably, a further memory module (23) is arranged after the last module (213) of the modules (210, 211, 212, 213) connected in series. The memory module can receive the output variable of the last module (213) as an input variable and can receive at least one output variable of the preceding filter module as an input variable by means of the connection (203). As a result, the filtered output variable of the preceding filter module can be stored in the last memory module. Alternatively, other arbitrarily predeterminable output variables of the modules (210, 211, 212, 213) or filter modules or memory modules (22, 23) can also be stored in the last memory module.

[0035] Figure 3A schematic diagram of an embodiment of a filter module (22) is shown. In this embodiment, the filter module (22) includes a normalization module (31), which is connected in series with the activation module (32) and the LSTM module (33). Alternatively, the filter module (22) can also include a Kalman filter. The normalization module (31) is configured to normalize the input variables of the filter module (22) by a normalization operation, in particular the largest variable included in the input variables of the filter module (22) is used so that the input variables are divided by this, so that the largest variable included in the input variables of the filter module (22) is normalized to one. Other normalization operations are also conceivable, such as normalization of the standard deviation of the input variables of the normalization module (31) and / or mean value cleaning. The following activation module (32) is preferably a module that outputs an output variable once the input variable of the activation module (32) exceeds a certain value. It is also conceivable that the activation module (32) includes a function, preferably a ReLu function (English: Rectifed Linear Unit function) or a S-type function (Sigmoid function), and the activation module (32) can determine the output variable based on the input variable of the activation module (32) and the function. The LSTM module (33) can be based on the intermediate variable (z t-1 ) and the output variable of the activation module (32) to determine the output variable and the latest intermediate variable (z t ).

[0036] In the embodiment of the filter module (22), the LSTM module (33) is used to perform a temporal filtering of an output variable of a preceding module of the filter module (22). t-1 ) has the result that robustness can be achieved, in particular with respect to noisy or ambiguous output variables from the detection unit (12). It is particularly preferred that the temporally preceding intermediate variable (z t-1) characterizes the temporal development of the output variable of the predeterminable module (210). Because by taking into account the temporal development of the output variable of the predeterminable module (210), it is possible to distinguish the inaccuracy or ambiguity of the output variable of the detection unit (12) or to achieve a lower vulnerability to the inaccuracy or ambiguity.

[0037] If the input variables of the system (3) are multidimensional, in particular include images or video sequences, the LSTM module (33) is preferably a convolutional LSTM module.

[0038] Figure 4 An advantageous embodiment of the filter module (22) is shown. This embodiment of the filter module (22) is similar to Figure 3 The difference from the embodiment in is that a jumper connection (401) is used. The jumper connection (401) connects the input of the filter module (22) to the output of the filter module (22). The input variable is added to the determined output variable of the filter module (22) via the jumper connection (401). It should be noted that in this embodiment, the dimensions of the input variable of the filter module (22) and the determined output variable of the LSTM module (33) must be the same.

[0039] Figure 5 An alternative embodiment of the filter module (22) is shown. Figure 4 The difference in the embodiment is that a decoding module (50) is connected between the LSTM module (33) and before the connection of the bridging connection (401) to the output of the filter module (22). The decoding module (50) decodes the output variable of the LSTM module (33) in such a way that the dimension of the forwarded input variable of the filter module (22) is the same as the determined output variable of the filter module (22).

[0040] Figure 6 A schematic diagram shows an embodiment of a method (60) for detecting an object by means of a system (13).

[0041] The method (60) begins with step 61. In step 61, at least one input variable is provided to the system (13) by means of the detection unit (12).

[0042] In the next step 62, starting with the first module (210) of the modules (210, 211, 212, 213) connected in series, the output variable of each module is determined step by step along the modules (210, 211, 212, 213) connected in series according to the input variables of the corresponding module and the parameterization of the corresponding module. If a filter module and a storage module are connected between two predeterminable modules (210, 211), the output variables of the filter module and the storage module (22, 23) are determined step by step after the output variable of the first module (210) is determined. The output variable of the storage module (23) is then used to continue to determine the output variables of the corresponding modules step by step along the modules (210, 211, 212, 213).

[0043] After step 62 has been completed, step 63 follows. In step 63, the system (13) is trained so that it can be used for detecting objects. During the training of the system (13), the parameterization of the modules (210, 211, 212, 213) is determined as a function of the input variables provided by the system (13) and the output variables determined from step 62, in particular using an optimization method such as a gradient descent method. Preferably, all parameterizations of the modules (210, 211, 212, 213) and of each filter module (22) used are determined with the aid of the optimization method. Optionally, step 62 can be performed repeatedly or for a plurality of input variables provided in order to perform the training of the system (13) more accurately.

[0044] If in step 63 the system (13) has been trained and can be used for detecting an object, then step 64 follows. In step 64, the detection unit (12) provides input variables for the system (13), which, for example, characterize the environment of the detection unit (12). The system (13) determines output variables step by step as a function of the input variables. The object is then detected with the aid of the detection module (14) as a function of the determined output variables of the system (13). Optionally, in step 64, the vehicle (11) is controlled as a function of the detected object.

[0045] In another embodiment of the method (60), it is conceivable that after the termination of step 64, the method (60) is cyclically started with step 61, if, for example, the system (13) is to be set up for another application. Alternatively, after the termination of step 64, step 64 can be restarted.

Claims

1. A method (60) for detecting an object by means of a system (13), wherein the system (13) comprises a plurality of modules (210, 211, 212, 213) connected in series one after the other, wherein each module comprises a deep neural network having at least one layer, wherein the at least one layer comprises neurons, wherein at least one connection between a predeterminable module (210) and its immediately following module (211) is respectively made by means of filter modules and memory modules (22, 23) connected in series, wherein the first module (210) of the modules (210, 211, 212, 213) connected in series is connected to a detection unit (12) and the last module (213) of the modules (210, 211, 212, 213) connected in series is connected to a detection module (14), the method comprising: - providing an input variable for the first module (210) of the modules connected in series by means of the detection unit (12), - starting with the first module (210) of the modules (210, 211, 212, 213) connected in series, the output variables of each module are determined step by step along the modules (210, 211, 212, 213) connected in series according to the input variables of the corresponding modules and the parameterization of the corresponding modules, wherein the output variable of the predefinable module (210) is filtered by means of the filter module (22), The filter module (22) is configured to filter the output variable of the predeterminable module and a temporally preceding intermediate variable (z t-1 ) to determine the output parameters, The filter module (22) selects a temporally preceding output variable and a temporally preceding intermediate variable (z t-1 ) to determine the latest intermediate parameter (z t ), The intermediate parameter (z t-1 ) has been determined by means of the filter module (22) based on the temporally preceding output variable of the predeterminable module (210), The output variable of the filter module (22) and at least one input variable of one of the preceding modules of the predeterminable module (210) are combined into a memory variable by means of the memory module (23) and provided as an input variable for a subsequent module (211).

2. The method according to claim 1, wherein: The filter module (22) is based on the intermediate variable (z t-1 ) to filter the output variable of the predeterminable module (210), wherein the temporally preceding intermediate variable (z t-1 ) characterizes the temporal development of the output variable of the predeterminable module (210).

3. A method according to any one of the preceding claims, wherein: The filter module (22) comprises a normalization module (31), an activation module (32) and an LSTM module (33), wherein the temporally latest intermediate variable (z_t) is converted by means of the LSTM module (33) according to the input variable of the LSTM module (33) and the temporally preceding intermediate variable (z_t) t-1 ) is determined.

4. The method according to claim 3, wherein: The filter module (22) additionally has a jumper connection (401), wherein when filtering the output variable of the predeterminable module (210) by means of the filter module (22), the input variable of the filter module (22) is forwarded to the output of the filter module (22) by means of the jumper connection (401), and at the output of the filter module (22), the input variable of the filter module (22) is also added to the output variable of the filter module (22).

5. A method according to claim 4, wherein the filter module (22) additionally includes a decoding module (50) and the decoding module (50) connects the LSTM module (33) to the output end of the filter module (22), wherein when the output parameter of the predefined module (210) is filtered by means of the filter module (22), the output parameter of the LSTM module (33) is decoded by means of the decoding module (50), so that the output parameter of the filter module (22) has the same number of dimensions as the input parameter of the filter module (22) forwarded by means of the jumper connection (401).

6. The method according to claim 1 or 2, wherein: The connection between the last module (213) of the modules (210, 211, 212, 213) connected in series and the detection module (14) is carried out by means of a third memory module, wherein the third memory module combines the output parameter of the last module (213) and at least one output parameter of a filter module connecting two consecutive modules (212, 213) of the modules (210, 211, 212, 213) connected in series into a memory parameter.

7. The method according to claim 1 or 2, wherein: An object is detected by means of a detection module (14) at the output of the last module (213) of the modules (210, 211, 212, 213) connected in series, wherein an at least semi-autonomous machine is controlled as a function of the detected object.

8. A computer program product comprising machine-readable instructions which, when executed on one or more computers, cause the one or more computers to implement the method according to any one of claims 1 to 7.

9. A machine-readable memory module on which a computer program product according to claim 8 is stored. 10 . Device for detecting an object, the device being designed to carry out the method according to claim 1 .