Method for assisting in controlling vehicle using improved image fusion information

Abstract

The present application relates to a method for assisting in controlling a vehicle using improved image fusion information, the method comprising: during an assisted driving function activation period of the vehicle; performing interpolation processing on pixel data about the following target object in the acquired image data about the environment in front of the vehicle: the resolution of the pixel data about the target object is lower than a preset resolution threshold (S1); and determining image fusion information about the target on the basis of the interpolated image data, and controlling an aided driving function of the vehicle on the basis of at least the determined image fusion information (S2). According to the invention, the accuracy and smoothness of the contour curve of the target object are improved, and then the pixel resolution of the target object is improved, so that the vehicle can recognize a long-distance or small-size target object earlier and more accurately. The safety and the performability of the auxiliary driving function with the image fusion information as the control basis are effectively improved, and then the maximum speed limit of the vehicle can be improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a method for assisting in the control of a vehicle using improved image fusion information, an electronic device, a vehicle including the electronic device according to this application, and a computer program product for at least assisting in implementing the steps of the method according to this application. Background Technology

[0002] The implementation of various driver assistance functions in vehicles relies on the detection information of the driving environment by various onboard sensors. In particular, the perception fusion information obtained from the driving environment image data collected by onboard cameras plays an important role in the identification of targets in front of the vehicle. However, when the target is far away from the vehicle or the target is small, the pixel resolution of the target in the collected driving environment image data is usually low. This will greatly affect the accuracy of the perception fusion information about the target, and thus affect the feasibility and safety of the related driver assistance functions.

[0003] Therefore, how to control the vehicle's driver assistance functions using low pixel resolution has become a technical challenge that needs to be solved. Summary of the Invention

[0004] The purpose of this application is to provide a method for assisting in the control of a vehicle using improved image fusion information, an electronic device, a vehicle including the electronic device according to this application, and a computer program product, to at least partially solve the problems in the prior art.

[0005] According to a first aspect of this application, a method for assisting vehicle control using improved image fusion information is provided, the method comprising:

[0006] - During the activation of the vehicle's driver assistance function, pixel data of the acquired image data regarding the environment in front of the vehicle are interpolated for targets whose resolution is lower than a preset resolution threshold; and

[0007] - Determine image fusion information about the target object based on the interpolated image data, and control the vehicle's driver assistance functions based at least on the determined image fusion information.

[0008] The core concept of this application is that, during the activation of the vehicle's assisted driving function, interpolation processing is performed on the pixel data of targets with low pixel resolution in the collected environmental image data. This improves the accuracy and smoothness of the target's contour curve, thereby increasing the pixel resolution of the target. As a result, richer and more accurate image fusion information of the target can be obtained based on the interpolated image data. Thus, the vehicle can identify distant or small-sized targets earlier and more accurately, effectively improving the safety and feasibility of assisted driving functions based on image fusion information control, and further increasing the vehicle's maximum speed limit.

[0009] According to an optional embodiment of this application, the desired trajectory of a vehicle can be adjusted based on image fusion information and road topology information about the target object, and the vehicle's steering, lane-changing, and / or braking operations can be controlled based on the adjusted desired trajectory to avoid collision with the target object.

[0010] According to another optional embodiment of this application, the interpolation process can be performed by one or more of the following methods: linear interpolation, cubic interpolation, and bicubic interpolation, etc.

[0011] According to another optional embodiment of this application, the horizontal pixel data of the target object in the image data can be processed by linear interpolation, and the vertical pixel data of the target object can be processed by cubic interpolation, so as to improve the pixel resolution of the target object.

[0012] According to another optional embodiment of this application, the horizontal and vertical pixel data of the image data relating to the target object can be processed by bicubic interpolation or linear interpolation to improve the pixel resolution relating to the target object.

[0013] According to another optional embodiment of this application, the driver assistance functions include, for example, city navigation function, and / or adaptive cruise function, and / or traffic jam assist function, and / or integrated cruise assist function, and / or lane centering control assist function, and / or automatic emergency braking function, etc.

[0014] According to a second aspect of this application, an electronic device is provided, the electronic device comprising:

[0015] -At least one processor; and

[0016] - A memory storing program instructions executable by the at least one processor, which, when executed by the at least one processor, implement the method according to this application.

[0017] According to another optional embodiment of this application, the electronic device may include an in-vehicle camera and / or a domain controller, wherein the in-vehicle camera particularly includes a forward-facing camera.

[0018] According to a third aspect of this application, a vehicle is provided, the vehicle including an electronic device according to this application.

[0019] According to a fourth aspect of this application, a computer program product, such as a computer-readable program carrier, is provided, comprising or storing computer program instructions that, when executed by a processor, at least assist in implementing the steps of the method described in this application. Attached Figure Description

[0020] The principles, features, and advantages of this application can be better understood by describing it in more detail below with reference to the accompanying drawings. The drawings show:

[0021] Figure 1 A flowchart illustrating a method for assisting vehicle control using improved image fusion information according to an exemplary embodiment of this application is shown.

[0022] Figure 2 A schematic diagram of a driving scenario according to an exemplary embodiment of this application is shown;

[0023] Figure 3 A schematic diagram illustrating an interpolation processing method for pixel data of a target object according to an exemplary embodiment of this application;

[0024] Figure 4 A schematic diagram of a driving scenario according to another exemplary embodiment of this application is shown; and

[0025] Figure 5 A schematic diagram of a vehicle according to an exemplary embodiment of this application is shown. Detailed Implementation

[0026] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit the scope of protection of this application.

[0027] Figure 1 A flowchart illustrating a method for assisting vehicle control using improved image fusion information according to an exemplary embodiment of this application is shown. The following exemplary embodiments describe the method according to this application in more detail.

[0028] like Figure 1As shown, the method may include steps S1 and S2. In step S1, during the activation of the vehicle's assisted driving function, pixel data of the collected image data about the environment in front of the vehicle regarding the following target objects can be interpolated: the pixel resolution of the target object is lower than a preset resolution threshold. In the embodiments of this application, the vehicle 1 may be equipped with different types of assisted driving functions according to different assisted driving system design architectures, especially L3 or higher level assisted driving functions, such as city navigation function designed for all road segments, and / or adaptive cruise function designed for closed highways or urban expressways, and / or traffic jam assist function designed for low-speed driving (below 60km / h) traffic congestion scenarios, and / or integrated cruise assist function designed for medium-high speed driving scenarios (above 60km / h), and / or lane centering control function designed for the entire speed range, and / or automatic emergency braking function, etc. Figure 2 This diagram illustrates a driving scenario according to an exemplary embodiment of this application. As a vehicle 1 with its assisted driving function activated approaches a target 2 from a distance, image data of the environment in front of the vehicle can be acquired via an onboard camera, particularly a forward-facing camera, and point cloud data of the environment in front of the vehicle can be acquired via millimeter-wave radar. Based on the image data, a target 2 within a pre-given distance range in front of the vehicle 1 can be identified. The target 2 includes, for example, motor vehicles such as two-wheeled vehicles, four-wheeled vehicles, and / or other multi-wheeled vehicles, as well as pedestrians, non-motorized vehicles, and / or obstacles. For autonomous driving systems, especially Level 3 and above, the reliability and robustness of target detection results are crucial in all driving scenarios. Therefore, the detection results of the target by millimeter-wave radar alone are insufficient; the target must also be detected by the onboard camera. However, when the target 2 is far from the vehicle (e.g., at 200m) or the target is small, the pixel resolution of the target in the acquired camera image data is lower than a preset resolution threshold. Figure 3 This diagram illustrates an interpolation process for pixel data of a target object according to an exemplary embodiment of this application. Four original pixel data points P1, P2, P3, and P4 for the target object 2 are marked with solid circles. Here, interpolation processing can be performed on the pixel data of the target object 2, and the interpolation processing can be performed in one or more ways, such as linear interpolation, cubic interpolation, and bicubic interpolation.

[0029] For example, pixel data related to the target object in the image data can be processed by linear interpolation to improve the pixel resolution related to the target object 2. Here, the average of the horizontal axis coordinate value X1 of the first original pixel data P1 and the horizontal axis coordinate value X2 of the second original pixel data P2 can be obtained as the first linearly interpolated pixel data P. L1 x-axis coordinate value L1 Alternatively, the average of the ordinate Y1 of the first original pixel data P1 and the ordinate Y2 of the second original pixel data P2 can be used as the first linear interpolation pixel data P. L1 The vertical axis coordinate value Y L1 Thus, the first linearly interpolated pixel data P obtained through linear interpolation is obtained. L1 It is marked with a hollow circle. Similarly, the second linear interpolated pixel data P can be obtained using the second original pixel data P2 and the third original pixel data P3. L2 And using the third original pixel data P3 and the fourth original pixel data P4 to obtain the third linear interpolated pixel data P L3 The four original pixel data P1, P2, P3, P4 and three linearly interpolated pixel data P L1 P L2 P L3 Connecting the dots with dashed lines yields a first composite contour curve for the target object 2. This first composite contour curve contains a much larger amount of pixel data than the original pixel data, thereby improving the pixel resolution for the target object 2.

[0030] For example, the horizontal pixel data of the image data relating to the target object 2 can be processed using linear interpolation, and the vertical pixel data relating to the target object 2 can be processed using cubic interpolation, thereby improving the pixel resolution relating to the target object. Exemplarily, the average of the horizontal coordinate value X1 of the first original pixel data P1 and the horizontal coordinate value X2 of the second original pixel data P2 can be calculated as the first hybrid interpolated pixel data P. L1 x-axis coordinate value B1 Then use the obtained x-axis coordinate value X B1 The first hybrid interpolation pixel data P is obtained using the following cubic interpolation formula. B1 The vertical axis coordinate value Y B1 :

[0031] Y = a * X 3 +b*X 2 +c*X+d,

[0032] Where a represents the coefficient of the cubic term in the cubic interpolation formula, b represents the coefficient of the quadratic term, c represents the coefficient of the linear term, and d represents the constant coefficient. These four polynomial coefficients a, b, c, and d can be obtained by fitting the coordinates of the four original pixel data points P1, P2, P3, and P4. Similarly, the second hybrid interpolated pixel data P can be obtained. B2 x-axis coordinate value B2 and the vertical axis coordinate value Y B2 and the third hybrid interpolated pixel data P B3 x-axis coordinate value B3 and the vertical axis coordinate value Y B3 The four original pixel data P1, P2, P3, P4 and the three mixed interpolated pixel data P are combined. B1 P B2 P B3 Connecting these points with solid lines yields a second composite contour curve for target object 2. This second composite contour curve contains significantly more pixel data than the original pixel data, thereby improving the pixel resolution for target object 2. Compared to the first composite contour curve, the smoothness of the second composite contour curve is greatly improved, thus more accurately reproducing the detailed contour information of target object 2.

[0033] For example, bicubic interpolation can be used to process the horizontal and vertical pixel data of the target object in the image data to improve the pixel resolution of the target object 2. For the 16 original pixels in a 4x4 column array of the target object 2, an intermediate result is first calculated along the horizontal axis using the aforementioned cubic interpolation method for each row of four pixels. Then, bicubic interpolation pixel data is calculated along the vertical axis using the same cubic interpolation method. Compared to the first and second synthesized contour curves, the accuracy and smoothness of the third synthesized contour curve obtained based on these original pixel data and the bicubic interpolated pixel data are further improved, thereby more accurately restoring the detailed contour information of the target object 2.

[0034] In step S2, image fusion information about the target object 2 can be determined based on the interpolated image data, and the assisted driving functions of vehicle 1 can be controlled at least based on the determined image fusion information. Here, the desired trajectory of vehicle 1 can be adjusted based on the image fusion information and road topology information about the target object, and vehicle 1 can be controlled based on the adjusted desired trajectory, particularly controlling the steering, and / or lane-changing, and / or braking operations of vehicle 1 to avoid collisions with the target object 2. Figure 2In the exemplary driving scenario shown, during the activation of the adaptive cruise control function of vehicle 1, based on interpolated image data, it can be determined that a stationary obstacle 2 exists, for example, 200m away from the vehicle. The target lane of vehicle 1 can be switched to the adjacent lane on the left, and the desired trajectory of vehicle 1 can be adjusted accordingly, thereby controlling vehicle 1 to change lanes to the left lane in advance. Figure 4 The diagram illustrates a driving scenario according to another exemplary embodiment of this application. As the vehicle 1 with the automatic emergency braking function activated approaches the target object 2 from a distance, it can be determined based on the interpolated image data that there is a stationary obstacle 200m away from the vehicle. It is also identified that multiple vehicles 3 are traveling at a small following distance in the adjacent lane, which makes it impossible for the vehicle 1 to have enough space to change lanes. Therefore, the vehicle 1 can be controlled to perform emergency braking in a timely manner.

[0035] According to embodiments of this application, during the activation of the vehicle's assisted driving function, pixel data of targets with low pixel resolution in the collected environmental image data are interpolated, thereby improving the accuracy and smoothness of the target's contour curve, and further improving the pixel resolution of the target. This allows for richer and more accurate image fusion information of the target to be obtained based on the interpolated image data, enabling the vehicle to identify distant or small-sized targets earlier and more accurately. This effectively improves the safety and feasibility of the assisted driving function based on image fusion information, and can also increase the vehicle's maximum speed limit.

[0036] In addition, it should be noted that the step numbers described herein do not necessarily represent the order of steps, but are merely a reference numeral. The order may be changed depending on the specific circumstances, as long as the technical objective of this application can be achieved.

[0037] Figure 5 A schematic diagram of a vehicle according to an exemplary embodiment of this application is shown. Figure 5 As shown, the vehicle 1 may include an electronic device 10, which may include:

[0038] -At least one processor 11; and

[0039] - Memory 12, in which program instructions are stored that can be executed by the at least one processor 11, and when the program instructions are executed by the at least one processor 11, implement the method according to the present application.

[0040] Optionally, the electronic device 10 may include an in-vehicle camera and / or a domain controller, wherein the in-vehicle camera particularly includes a forward-facing camera.

[0041] It should be understood that the terms “first,” “second,” “third,” etc., used in this document are for descriptive purposes only and should not be construed as indicating or implying relative importance, nor should they be construed as implicitly specifying the number of technical features indicated.

[0042] If an embodiment includes an "and / or" association between a first feature and a second feature, it should be interpreted as follows: according to one implementation, the embodiment has not only the first feature but also the second feature; according to another implementation, the embodiment has either only the first feature or only the second feature.

[0043] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of this application, even when only a single embodiment is described with respect to a particular feature. The feature examples provided in this application are intended for illustrative purposes and not for limitation, unless otherwise stated. In practice, multiple features may be combined with each other as needed and where technically feasible. Various substitutions, modifications, and alterations are also conceived without departing from the spirit and scope of this application.

Claims

1. A method for assisting vehicle control using improved image fusion information, the method comprising: During the activation of the vehicle's driver assistance function, the pixel data of the collected image data about the environment in front of the vehicle are interpolated for the following targets: the resolution of the pixel data of the target is lower than a preset resolution threshold; as well as Based on the interpolated image data, image fusion information about the target object is determined, and the vehicle's driver assistance functions are controlled at least based on the determined image fusion information.

2. The method according to claim 1, wherein, The desired trajectory of the vehicle (1) is adjusted based on image fusion information and road topology information about the target object, and the steering operation, and / or lane change operation, and / or braking operation of the vehicle (1) are controlled based on the adjusted desired trajectory to avoid collision with the target object.

3. The method according to any one of the preceding claims, wherein, The interpolation process is performed using one or more of the following methods: linear interpolation, cubic interpolation, and bicubic interpolation.

4. The method according to any one of the preceding claims, wherein, The horizontal pixel data of the target object in the image data is processed by linear interpolation, and the vertical pixel data of the target object is processed by cubic interpolation to improve the pixel resolution of the target object.

5. The method according to any one of the preceding claims, wherein, The horizontal and vertical pixel data of the target object in the image data are processed by bicubic interpolation or linear interpolation to improve the pixel resolution of the target object.

6. The method according to any one of the preceding claims, wherein, The driver assistance functions include city navigation, and / or adaptive cruise control, and / or traffic jam assist, and / or integrated cruise assist, and / or lane centering control assist, and / or automatic emergency braking.

7. An electronic device (10), said electronic device (10) comprising: At least one processor (11); and A memory (12) storing program instructions executable by the at least one processor (11) to implement the method according to any one of claims 1 to 6 when the program instructions are executed by the at least one processor (11).

8. The electronic device (10) according to claim 7, wherein, The electronic device (10) includes an in-vehicle camera and / or a domain controller, wherein the in-vehicle camera particularly includes a forward-facing camera.

9. A vehicle (1) comprising an electronic device (10) according to claim 7 or 8.

10. A computer program product, such as a computer-readable program carrier, comprising or storing computer program instructions that, when executed by a processor, at least auxiliaryly implement the steps of the method according to any one of claims 1 to 6.

Images