Method for determining output data on the basis of sensor data detected by an optical sensor system and control module for a motor vehicle
Patent Information
- Application Number
- CN202610521446.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-04-28
- Filing Date
- 2026-04-20
- Publication Date
- 2026-10-30
Smart Images
Figure CN123016369A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for determining output data based on sensor data detected by an optical sensor system, as described in the preamble of claim 1. Background Technology
[0002] For certain applications, it is necessary or desirable to extract specific information contained in the sensor data detected by the optical sensor system as output data, or to derive specific output data from it, within the acquisition range detected by the optical sensor system, and to provide this output data for further processing.
[0003] This is particularly applicable to driver assistance systems in motor vehicles used to control vehicle operation under specific driving conditions or during specific movements, thereby assisting or reducing the driver's workload. These systems typically require and process information about the surrounding traffic space to function effectively. For example, this applies to lane keeping assist, overtaking assist, cornering assist, traffic jam assist, parking assist, or distance assist for longitudinal vehicle control. Therefore, these systems can be used in assisted driving modes, automatic driving modes, or autonomous driving modes of the motor vehicle. Summary of the Invention
[0004] The objective of this invention is to provide a method for determining output data based on sensor data detected by an optical sensor system, wherein different output data are generated and output in a simple and advantageous manner by processing the sensor data of the optical sensor system, and with high flexibility.
[0005] This task is accomplished by a method for determining output data based on sensor data detected by an optical sensor system, which has the features described in claim 1.
[0006] The advantageous improvement to the method according to the invention, and a control module for a vehicle performing the method, are integral parts of the other claims.
[0007] In the method of the present invention, in successive method segments, the same image sequence detected by the optical sensor system of the motor vehicle, which is related to the reference surface by means of image transformation processing, is processed in each method segment. The same image sequence consists of two environmental images and is processed by a single neural network using the relative position information of the optical sensor system, thereby generating different output parameters as independent output data when using modified optical flow.
[0008] Specifically, in the method of the present invention, in order to determine output data based on sensor data detected by the optical sensor system, in successive method segments during vehicle movement, a camera / camera arranged in the vehicle detects camera images as optical sensor data, said camera images being detected as two-dimensional grid images composed of grid cells. For this purpose, the optical sensor system or camera can move in different directions, particularly forward and / or backward and / or laterally.
[0009] In each method segment, during vehicle travel, two environmental images from the same image sequence are detected sequentially in a staggered manner as two-dimensional raster images composed of raster units or image points (pixels). One of the two environmental images from the image sequence, or a feature image obtained by processing one of the two environmental images from the image sequence, is used as the original image or starting image. This original image or starting image is then transformed or converted relative to a reference plane into a transformed image using geometric homography transformation as an image transformation. This transformed image serves as the transformed raster image or transformed feature image. By using this two-dimensional geometric homography transformation or image transformation, different perspectives can be created for two raster images or two feature images of the same image sequence. The method is as follows: one of the two raster images or one of the two feature images of the image sequence that are detected in sequence is transformed into a transformed raster image or a transformed feature image, so that the raster cells or pixels of the transformed raster image or the transformed feature image appear to be "looking" at a reference plane that is a preset for the untransformed raster image or the untransformed feature image. The untransformed raster image or the untransformed feature image represents the reference image, and therefore represents the reference raster image or the reference feature image.
[0010] By employing geometric homography transformation as an image transformation, a virtual viewpoint pointing towards a reference plane is obtained based on two environmental images of the same image sequence consisting of two sequentially detected raster images. This ensures that the positions of the image points of the transformed image on the reference plane are identical to those of the temporally staggered reference images. Therefore, since the image points on the reference plane remain unchanged during the same image sequence consisting of two sequentially detected raster images or two feature images, significant stabilization of the image sequence is achieved, simplifying motion processing between two raster images or two feature images within the same image sequence. This further simplifies the processing of the raster images or feature images of the image sequence. In particular, this also allows for significantly simplified determination of optical flow or modified optical flow based on corresponding image points in the two raster images or two feature images of the same image sequence.
[0011] As described above, this motion simplification is achieved by using geometric homography transformation as an image transformation, applied to the original or starting image of the same image sequence of raster images, or applied to the original or starting image as a feature image obtained by processing one of the two raster images or environment images of the same image sequence.
[0012] Here, the reference plane is specifically chosen as the road surface on which the vehicle travels, because after applying geometric homography transformation, the image points on the road surface will not move, and therefore can be used as a constant reference quantity.
[0013] Within the framework of geometric homography transformation as an image transformation, specifically in a given image sequence, the first original image of two original images is converted into a transformed image, and the second original image is used as an untransformed reference image. However, the two original images in a given image sequence can also be assigned in the opposite way. That is, in this case, in a given image sequence, the second original image is converted into a transformed image, and the first original image of that given image sequence is used as an untransformed reference image. This assignment relationship between the transformed image and the reference image for the two original images of the same image sequence remains unchanged in all successive method segments of the present invention.
[0014] Therefore, this processing method is equally applicable whether geometric homography transformation is applied to one of two raster images detected sequentially in the same image sequence, or to one of two feature images obtained by processing raster images in the same image sequence.
[0015] The image sequence consists of two temporally staggered environmental images detected sequentially as two-dimensional raster images composed of raster units or pixel points during vehicle operation, and is processed by a neural network, preferably a neural network with a pyramid network architecture having multiple processing layers.
[0016] When applying geometric homography transformation to one of two raster images or environmental images in the same image sequence, or when applying geometric homography transformation to one of two feature images in the same image sequence, the neural network, preferably having a pyramidal network architecture, is also input with: spatial motion data between temporally staggered detections of two raster images in the same image sequence by the optical sensor system, determined based on vehicle motion data acquired within the odometer framework, and the relative position of the optical sensor system relative to a reference plane used for calibration of the optical sensor system when detecting each raster image in the same image sequence.
[0017] Therefore, the position of the optical sensor system relative to the reference plane is given in an approximate form, specifically in four dimensions using three spatial coordinates plus the vehicle's yaw angle, or in three dimensions using three spatial coordinates, or in estimated, and therefore potentially erroneous, position information using three spatial coordinates and three spatial angles. In particular, the approximate position information of the optical sensor system is determined based on the detection of the reference plane's normal vector and the height of each grid image's pixel relative to the reference plane.
[0018] Within the scope of external calibration of such an optical sensor system, it is particularly important to determine the mounting position of the optical sensor system relative to the reference plane within the vehicle. In addition to this external calibration, internal calibration can also be performed based on the optical characteristics of the optical sensor system, for example, by determining the mounting configuration of the optical sensor within the system.
[0019] The modified optical flow is processed and output by a neural network with a preferred pyramidal network architecture, based on transformed and reference images of the same image sequence, i.e., transformed raster images and reference raster images of the same image sequence, or transformed feature images and reference feature images of the same image sequence. The processing of the input data is therefore performed by a single neural network with a pyramidal network architecture, which determines the modified optical flow as: virtual motion between two raster images or two feature images of the same image sequence, derived from two raster images or two feature images of the same image sequence.
[0020] The neural network cumulatively or alternatively outputs the following parameters as output data: corrected optical flow, geometric parameters, the relative position of the optical sensor system between two raster images in the same image sequence (preferably represented in six dimensions), and the normal vector of each pixel of the raster image.
[0021] The structural parameters used to calculate specific other output parameters are represented as a three-dimensional representation, specifically determined based on the homography matrix used when applying geometric homography transformations. In particular, these structural parameters are measures of the height of image points relative to the reference plane and can be used together with the depth of each image point relative to the optical center of the optical system, especially for the three-dimensional reconstruction of the reference image.
[0022] Precise, specifically six-dimensional, relative positional information of an optical sensor system between two environmental images in the same image sequence, as well as the spatial orientation (spatial orientation) of the optical sensor system within the vehicle and its position relative to a reference plane between the two environmental images in the same image sequence, are represented in a six-dimensional format. This includes, in addition to three spatial coordinates, yaw angle, roll angle, and pitch angle. Specifically, the yaw angle is a measure of the optical sensor system's rotation about its vertical axis, the roll angle is a measure of its rotation about its longitudinal axis, and the pitch angle is a measure of its rotation about its transverse axis.
[0023] In a favorable improvement to this method, in order to transform one of the two original images of the same image sequence into a transformed image, a disparity transformation about a reference plane is performed as a geometric homography transformation using a homography matrix. That is, based on a geometric transformation using virtual disparity, the image points of one of the two original images of the same image sequence are moved accordingly using a homography matrix.
[0024] This homography matrix is created based on reference plane and vehicle odometer data as an abstract and simplified representation of the vehicle environment based on virtual parallax, and therefore contains geometric assumptions about the image points of the reference plane or road surface and two environmental images of the same image sequence.
[0025] Specifically, based on the virtual disparity transformation performed through the homography matrix as a homography transformation, structural parameters can be determined and output as needed within the neural network processing framework, or used for corresponding application scenarios and further applications.
[0026] In another advantageous improvement to this method, the transformed image is inversely transformed back to the initial original image using structural parameters, and the optical flow between two environmental images in the same image sequence is determined based on the corrected optical flow. This optical flow is a characteristic quantity characterizing vehicle motion and can be used in relevant application scenarios and for further applications.
[0027] In another advantageous improvement to this method, 3D reconstruction is performed on one of two environmental images from the same image sequence using structural parameters, particularly on a reference image from the same image sequence. Within this 3D reconstruction framework, the height and depth of the image points in the reference image are calculated based on the structural parameters and the normal vector relative to the reference plane. This 3D representation of the reference image can then be used for the corresponding application scenario and for further applications.
[0028] The normal vector of each pixel in the raster image, used as an output parameter, can be used for the 3D representation of two environmental images from the same image sequence, but it is also particularly useful for the calibration of optical systems, especially for calibrating cameras used in motor vehicles as optical systems. In the latter case of camera calibration, the normal vector from a specific pixel in one of the raster images, which is particularly suitable for this purpose, can be used.
[0029] By using a neural network with a pyramidal network architecture that has multiple layered processing layers, each processing layer sequentially processes the reference image and the transformed image of the same image sequence in each method segment. The reference image and the transformed image of the same image sequence are assigned different resolutions and different information contents depending on the processing layer.
[0030] Therefore, different attributes or features of the reference and transformed images are used in each processing layer with varying resolution and detail. Using a neural network with a pyramidal network architecture, in successive processing steps within the method segment, downsampling of the reference and transformed images of the same image sequence is first performed in subsequent progressively descending processing layers, with processing occurring in network layers where resolution gradually decreases and information content gradually increases. In subsequent progressively ascending processing layers, upsampling further provides the information generated by the neural network based on the continuously increasing resolution.
[0031] In other words, within each method segment, firstly, in the successively descending processing layers, the resolution of the reference image and transformed image of the same image sequence is progressively reduced through downsampling, and the information content of the reference image and transformed image of the same image sequence is progressively increased based on feature channels or image channels. The reference image and transformed image of the same image sequence are processed and fused accordingly in each processing layer. Subsequently, in the successively ascending processing layers, the content of the processed and fused reference image and transformed image of the same image sequence is progressively provided based on upsampling. Thus, in the processing and fusion of the reference image and transformed image of the same image sequence, the initial rough estimate is progressively refined as more and more input content and information are added. For example, during the downsampling process of the successively descending processing layers, the number of raster cells or pixels of the reference image and transformed image of the same image sequence under consideration is progressively halved, while simultaneously, the number of feature channels or image channels used for the reference image and transformed image of the same image sequence as information content is progressively increased.
[0032] As previously described, in this method, two raster images from the same image sequence can be processed not only as "images" themselves, but also as feature images derived from the raster images based on their content and feature attributes. To this end, an encoder analyzes the characteristics or features of the two raster images from the same image sequence and stores the relevant image characteristics or features of each raster image in the feature image as a so-called feature map. For example, the contour edges contained in each raster image from the same image sequence can be used as a relevant image characteristic or feature for each raster image. In successive processing steps of the method segment, a homography matrix for geometric homography transformation is applied to one of the two feature images at the corresponding processing layer, thereby transforming its characteristics or features accordingly.
[0033] At the corresponding processing layer, the fusion of the reference image and the transformed image is preferably accomplished through a correlation method, namely the so-called cross attention method. In particular, in the local cross attention method, the relationship between individual pixels in the reference image and the transformed image of the same image sequence in the local neighborhood is determined based on matrix operations, thereby particularly determining the similarity between pixels in the reference image and the transformed image of the same image sequence.
[0034] In order to control the method of the present invention and its various method segments, thereby controlling the method flow, a control module or control device may be provided to perform control over the optical sensor system and neural network in order to process and evaluate the optical sensor data.
[0035] The components used to perform the method of the present invention, particularly the control module or control device, neural network, and optical sensor system, can be implemented as independent components of the vehicle and networked within the vehicle. Alternatively, individual or all components of the control module or control device can be integrated into a common modular component within the vehicle.
[0036] The vehicle used in applying the method of the present invention can be a motor vehicle or automobile, such as a passenger car, commercial vehicle, motorcycle or bus.
[0037] The vehicle can operate in assisted driving mode and / or automatic driving mode and / or autonomous driving mode.
[0038] The present invention also includes combinations of features from the various embodiments. Therefore, the present invention also includes implementations that each have a combination of features from multiple embodiments, provided that these embodiments are not described as mutually exclusive.
[0039] Using the method of the present invention, a variety of output data can be extracted from sensor data acquired by a vehicle optical sensor system in a simple and advantageous manner with low overhead and high precision, and provided for further processing. Attached Figure Description
[0040] An embodiment of the present invention is described below. Wherein, Figure 1 A schematic block diagram showing the components and functional units used in the method of the present invention is provided. Figure 2 A view showing an image representation used in a method segment according to a first embodiment of the method of the present invention. Figure 3 A schematic diagram illustrating the method flow in a method segment is shown, according to a second embodiment of the method of the present invention.
[0041] The following embodiments are preferred embodiments of the present invention. In each embodiment, the described components constitute independent and separately conceivable features of the present invention, and these features can also independently further develop the present invention. Therefore, they can be considered as components of the present invention individually, or they can appear in combinations different from those shown. Furthermore, the embodiments can be supplemented by other features already described in the present invention.
[0042] Components with the same function in the figure are represented by the same reference numerals.
[0043] List of reference numerals
[0044] 1. Input Data
[0045] 2 Neural Networks
[0046] 3 Output Data
[0047] 4. Optical sensor system / camera
[0048] 5 vehicles
[0049] 6. Control Module
[0050] 10 image sequences
[0051] 11 First raster image
[0052] 12 Second raster image
[0053] 13 Original Images
[0054] 14 Transform the image
[0055] 15 Reference Images
[0056] 16 Virtual disparity vectors
[0057] 17 Virtual parallax lines
[0058] 18. Approximate camera position
[0059] 19. Calibration
[0060] 20. Homography Transformation
[0061] 21 Encoders
[0062] 22 Feature Images
[0063] 23 Processing Layer
[0064] 24 downsampling
[0065] 25 upsampling
[0066] 26 Transformation Steps
[0067] 27. Fusion Steps
[0068] 28 Additional Information Steps
[0069] 29 Reference Plane
[0070] 31 Structural Parameters
[0071] 32. Detailed camera locations
[0072] 33. Corrected optical flow
[0073] 34 optical flow
[0074] 35. Normal vector of each image point
[0075] 36 Extreme Points
[0076] 37 Homography Matrix Detailed Implementation
[0077] like Figure 1 As shown, the method of this invention utilizes a single neural network 2, which has a specific pyramidal network architecture and successive processing layers 23, for processing different input data 1. These input data 1 are processed step by step in the various processing layers 23 of the neural network 2 in successive method steps, thereby generating and outputting different output data 3 according to the application, and providing them to the corresponding application for further processing when needed.
[0078] After training the neural network 2 based on the training data used as input data 1 during the training phase, the method of the present invention can be applied during the driving of the vehicle 5. The vehicle 5 has a control module 6 that controls the neural network 2 of the vehicle 5 and an optical sensor system 4, for example, configured as a camera. To this end, during the driving of the vehicle 5, in successive method segments, the camera 4 of the vehicle 5 detects optical sensor data as two-dimensional raster images 11 and 12 within the acquisition range of the traffic space surrounding the vehicle 5. These two-dimensional raster images are composed of raster cells or pixels. These optical sensor data of raster cells or pixels, for example, in the form of color values or RGB values, can be converted into an information domain with information values. Thus, each raster cell of the raster images 11 and 12, in addition to having an index value that locates its position within the acquisition range, is assigned at least one feature information or attribute as a feature, thereby generating a corresponding feature image.
[0079] Specifically, in each method segment, two environmental images of the same image sequence 10, namely two-dimensional raster images 11 and 12, are detected sequentially in time, and two corresponding feature images can be derived from them. That is, in successive method segments, the two raster images 11 and 12 of the same image sequence 10, detected sequentially in time by the camera 4 of the vehicle 5 within a specific acquisition range, are fed as input data 1 to the neural network 2. The two raster images 11 and 12 of the same image sequence 10 are detected by the camera 4 of the vehicle 5 at a frequency of, for example, 25 Hz, i.e., a time interval of 40 ms. This means that the second raster image 12 of the same image sequence 10 is always detected by the camera 4 40 ms after the first raster image 11 of the same image sequence 10.
[0080] like Figure 2Specifically, according to a first embodiment of the present invention, one of the two raster images 11 and 12 of the same image sequence 10 is transformed from the original image 13 to the transformed image 14 using a geometric homography transformation 20 via a homography matrix 37, thus becoming a transformed raster image; while the other of the two raster images 11 and 12 of the same image sequence 10 is not transformed, and the original image 13 is used as a reference image 15, i.e., a reference raster image. Preferably, each second raster image 12 of the same image sequence 10 is used as a reference image 15, i.e., a reference raster image, and therefore is not transformed, so that the original image 13 remains unchanged; while each first raster image 11 of the same image sequence 10 is transformed from the original image 13 to the transformed image 14 using a homography matrix 37 (e.g., a 3×3 matrix), i.e., becoming a transformed raster image 14. By performing this geometric homography transformation 20 on the first raster image 11 of the same image sequence 10, even if the vehicle 5 detects two raster images 11 and 12 of the same image sequence 10 at two staggered times, the two raster images 11 and 12 of the same image sequence 10 still "look" at the same image point on the reference plane 29 that remains almost unchanged.
[0081] The geometric homography transformation 20, performed using the homography matrix 37 as a disparity transformation relative to the reference plane 29, in Figure 2 A more detailed illustration is shown in the lower right image. Here, in the transformed image 14, obtained as a transformed raster image through geometric homography transformation 20, the reference point of homography transformation 20 is shown as pole 36, the virtual disparity line 17 emanating from pole 36, and the virtual disparity vector 16. The virtual disparity vector 16 is a measure of motion between the stabilized raster images 11 and 12 of the same image sequence 10 and provides a basis for determining the geometric structure parameter 31 or transformation parameter.
[0082] Furthermore, in a specific method segment, the untransformed reference image 15 of the second raster image 12 of the same image sequence 10 is transformed into the transformed image 14 in a subsequent method segment and used as the transformed raster image of that subsequent method segment.
[0083] according to Figure 1 In addition to the two raster images 11 and 12 of the same image sequence 10 as input data 1, the neural network 2 will also be provided with: relative position data of the camera position 18 of the camera 4 during the detection of the two raster images 11 and 12 of the same image sequence 10, and calibration data related to the pose of the camera 4 for camera 4 calibration 19.
[0084] The relative position data of the camera position 18 of the camera 4 is determined based on the odometer data of the vehicle 5, and includes, for example, three spatial coordinates in three dimensions, or three spatial coordinates in four dimensions plus a yaw angle representing the rotation of the vehicle 5 about its vertical axis, as a rough and potentially erroneous approximation during the detection of two grid images 11, 12 in the same image sequence 10.
[0085] As calibration data for camera 4 calibration 19 and attitude determination, it is particularly important to determine the normal vector of reference plane 29 relative to camera 4, and the height of each image point of the two raster images 11 and 12 of the same image sequence 10 relative to reference plane 29.
[0086] The pyramid-shaped neural network 2 can be trained according to the application scenario and purpose of the method to provide output data 3 suitable for that application scenario and purpose, and then put into use accordingly. This output data 3, generated after processing the input data 1 by the pyramid-shaped neural network 2, can be provided selectively or cumulatively by the neural network 2.
[0087] For example, after processing two raster images 11 and 12 of the same image sequence 10, neural network 2 can generate the following output data 3 using calibration data for camera 4 calibration 19 and virtual disparity vector 16: geometric structure parameters 31 or transformation parameters, six-dimensional precise positioning data of the relative position 18 of camera 4, and corrected optical flow 33, derived from transformed image 14 and reference image 15, as a measure of motion between the two raster images 11 and 12 of the same image sequence 10. Specifically, the geometric structure parameters 31 or transformation parameters can be determined using the calibration data for calibration 19, and thus for the pose of camera 4, and the virtual disparity vector 16. The structure parameters 31 or transformation parameters can be used to reconstruct a three-dimensional environment image from the two-dimensional raster images 11 and 12 of the same image sequence 10 by calculating the height and depth of each raster point in the two raster images 11 and 12 of the same image sequence 10 based on the structure parameters 31 or transformation parameters. Furthermore, starting from the modified optical flow 33, the optical flow 34, which is the motion metric between two raster images 11 and 12 of the same image sequence 10, can be determined by using the structural parameter 31 to inversely transform the transformed image 14 back to the original image 13.
[0088] Figure 3 Several functional units of a neural network 2 according to a second embodiment of the present invention are schematically illustrated. Figure 3 The correspondence between the functional units of neural network 2 and the steps of each method shown is for illustrative purposes only, because the allocation, function, and task of these functional units of neural network 2 cannot be strictly distinguished, and a single functional unit may also participate in... Figure 3 Other methods and steps besides those shown.
[0089] The neural network 2, with a pyramidal network architecture having multiple hierarchical processing layers 23, has input data 1 that specifically includes: a second raster image 12 of the currently detected image sequence 10 as the original image 13, and a cached first raster image 11 of the same image sequence 10 as the original image 13. The encoder 21 extracts information or attributes contained in the two raster images 11 and 12 of the same image sequence 10, thus extracting features and assigning these derived features to the two raster images 11 and 12. These feature patterns in the two raster images 11 and 12 of the same image sequence 10 are stored in two feature images 22. In successive method steps, the two feature images 22 are processed separately in their respective original processing layers 23 at different scaling ratios. At this time, specific resolutions and information contents are assigned to the two feature images 22 of the same image sequence 10 on their respective processing layers 23, so that the two feature images 22 of the same image sequence 10 are processed in different processing layers 23 with different resolutions and different information contents. For example, within the framework of downsampling 24 in processing layer 23, the resolution of the two feature images 22 is reduced (e.g., halved successively) in each successive processing layer 23, while the information content of the two feature images 22 of the same image sequence 10, determined based on information channels, is increased in each successive processing layer 23. For example, the two feature images 22 of the same image sequence 10 are processed in two dimensions, at full resolution (height and width), and with 3 information channels in the first processing layer 23, i.e., each feature image 22 has 1024 × 1024 pixels. In the next processing layer 23, the two feature images 22 of the same image sequence 10 are processed at half the resolution (height and width) relative to the first processing layer 23 and with 16 information channels, i.e., each feature image 22 has 512 × 512 pixels. In the next processing layer 23 after that, the two feature images 22 of the same image sequence 10 are processed at a resolution of one-quarter of the height and width and with 32 information channels, i.e., each feature image 22 has 256 × 256 pixels, and so on.
[0090] For example, the neural network 2 may be configured with four such processing layers 23, in which the two feature images 22 are processed accordingly. In each processing layer 23, for example in the transformation step 26, the first feature image 22 of the same image sequence 10 is transformed into a transformed feature image, i.e., the transformed image 14, by using the homography matrix 37 through the homography transformation 20.
[0091] Subsequently, in each processing layer 23, in the fusion step 27, the transformed feature image, which is the transformed image 14, is fused with the second feature image, which is the reference image 15 (i.e., the reference feature image), in particular using the local cross-attention method as a special correlation method.
[0092] Next, in each processing layer 23, in the additional information processing step 28, based on the approximate position data of the camera position 18 of the input camera 4, additional information is used to lay the foundation for determining the geometric structure parameters 31.
[0093] The correspondence or correlation information of two feature images 22 of the same image sequence 10 obtained on the corresponding processing layer 23, and the additional information obtained on the corresponding processing layer 23 in the additional information processing step 28 for determining the geometric structure parameter 31, are progressively passed to the higher-level processing layer 23 in the upsampling 25 framework.
[0094] Finally, based on the results of one of the processing layers 23, especially the topmost processing layer 23, the neural network 2 provides corresponding output data 3 suitable for the application scenario of the method. For example, the neural network 2 can output structural parameters 31, optimized and therefore accurate relative position 32 data of the camera 4 in the vehicle 5, corrected optical flow 33, the optical flow 34 thus determined, and the normal vector 35 of each pixel of the environmental image as output data 3, and can be further processed, for example, for use by at least one assistance system of the vehicle to control the driving task of the vehicle.
[0095] Overall, these examples demonstrate how to flexibly determine the output information required for each application purpose in an advantageous manner based on optical sensor data detected by the vehicle's optical sensor system in the detection area of the traffic space around the vehicle during vehicle movement.
Claims
1. A method for determining output data (3) based on sensor data detected by an optical sensor system (4), wherein, The sensor data is detected by the optical sensor system (4) installed in the vehicle (5) in successive method segments during the vehicle's (5) operation. Its features are, In each method segment, two environmental images are detected in a time-staggered manner during the vehicle's (5) travel as two-dimensional raster images (11, 12) of the same image sequence (10). The original image (13), which is one of the two raster images (11, 12) of the same image sequence (10) or one of the two feature images (22) derived from the two raster images (11, 12) of the same image sequence (10), is transformed into a transformed image (14) associated with the reference plane (29) by means of geometric homography transformation (20), and the original image (13), which is the other of the two raster images (11, 12) of the same image sequence (10) or the other of the two feature images (22) derived from the two raster images (11, 12) of the same image sequence (10), is represented as an untransformed reference image (15). The reference image (15) and the transformed image (14) of the same image sequence (10) are processed by the neural network (2), wherein the motion data of the optical sensor system (4) between the time-staggered detections of two grid images (11, 12) of the same image sequence (10) are determined based on the motion data of the vehicle (5), and the relative position of the optical sensor system (4) relative to the reference plane (29) is determined for the calibration (19) of the optical sensor system (4) when detecting each grid image (11, 12) of the same image sequence (10); The modified optical flow (33) is determined by the neural network (2) by processing the reference image (15) and the transformed image (14) of the same image sequence (10); The neural network (2) outputs the following output data (3): the corrected optical flow (33), and / or geometric parameters (31), and / or the relative position of the optical sensor system (4) between two raster images (11, 12) in the same image sequence (10), and / or the normal vector (35) of each pixel of the raster images (11, 12) in the same image sequence (10).
2. The method according to claim 1, characterized in that, In order to convert the original image (13) into the transformed image (14), a disparity transformation about the reference plane (29) is performed by means of the homography matrix (37) as a geometric homography transformation (20).
3. The method according to claim 1 or 2, characterized in that, The optical flow (34) between two raster images (11, 12) of the same image sequence (10) is determined based on the modified optical flow (33).
4. The method according to any one of claims 1 to 3, characterized in that, The two feature images (22) of the same image sequence (10) are derived from the two raster images (11, 12) of the same image sequence (10) in the following manner: the features of the two raster images (11, 12) of the same image sequence (10) are analyzed by an encoder (21), and the relevant features of each raster image (11, 12) of the same image sequence (10) are stored in the corresponding feature image (22).
5. The method according to any one of claims 1 to 4, characterized in that, The optical sensor system (4) is calibrated (19) based on the normal vector of the reference plane (29) and the height values relative to the reference plane (29) determined by each pixel of the raster images (11, 12) for the same image sequence (10).
6. The method according to any one of claims 1 to 5, characterized in that, The reference image (15) is reconstructed using structural parameters (31).
7. The method according to any one of claims 1 to 6, characterized in that, In each method segment, the reference image (15) and the transformed image (14) of the same image sequence (10) are processed by a neural network (2) with a pyramidal network architecture including multiple processing layers (23).
8. The method according to claim 7, characterized in that, In each processing layer (23) of the neural network (2), different resolutions and different information contents are assigned to the reference image (15) and the transformed image (14) of the same image sequence (10).
9. The method according to claim 8, characterized in that, In each method segment, the resolution of the reference image (15) and the transformed image (14) of the same image sequence (10) is gradually reduced by downsampling (24) in the successive downsampling processing layer (23) of the neural network (2), and the information content of the reference image (15) and the transformed image (14) of the same image sequence (10) is gradually increased; then, the reference image (15) and the transformed image (14) of the same image sequence (10) are gradually fused in each processing layer (23) by upsampling (25) in the successive upsampling processing layer (23) of the neural network (2).
10. The method according to claim 9, characterized in that, In each processing layer (23) of the neural network (2), the reference image (15) and the transformed image (14) of the same image sequence (10) are fused (27) by means of the cross attention method.
11. A control module for a vehicle (5) having a neural network (2) and an optical sensor system (4), and for performing the method of any one of claims 1 to 10.