A method, device and computer equipment for detecting vanishing lines

Abstract

The present invention relates to a vanishing line detection method, device and computer equipment, wherein the method comprises: obtaining a previous frame image and a current frame image, and performing 2D feature extraction on the previous frame image and the current frame image respectively to obtain the previous frame 2D feature and the current frame 2D feature; performing feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain the previous frame BEV feature and the current frame BEV feature; performing feature fusion on the previous frame BEV feature and the current frame BEV feature to obtain the fused BEV feature; and obtaining the vanishing line height value and the vanishing line height change value according to the fused BEV feature. Through the present application, the problem of low accuracy of vanishing line detection in the related art is solved, and the technical effect of improving the accuracy of vanishing line detection is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a vanishing line detection method, device and computer equipment. Background Art

[0002] With the improvement of computer technology and sensor technology, autonomous driving assistance technology is gradually entering the commercialization stage. Various OEMs and suppliers are gradually building a more robust automatic assisted driving ecosystem. Autonomous driving assistance technology integrates data from multiple sensors (lidar, millimeter-wave radar, ultrasonic radar, camera, etc.) to automatically control the vehicle's acceleration, braking and steering systems. In some cases, it can even completely replace human drivers in vehicle operations. Autonomous driving assistance technology represents an important direction for the future development of automobiles, which can effectively reduce traffic accident rates and improve driving convenience.

[0003] Lane detection is an important part of automatic assisted driving, and vanishing line detection also plays an important role in the lane detection process. The effectiveness and stability of vanishing line detection help to accurately estimate the external parameters of the vehicle system such as pitch angle, which is of positive significance for subsequent tasks such as speed measurement and distance measurement.

[0004] Related technologies usually use the following methods to detect vanishing lines: vanishing line detection is performed by reconstructing the similarity of NCC values ​​in the road color space, but it is easily affected by the occlusion of image features; vanishing points are fitted through the geometric characteristics of parallel lines in the image, such as the straight lines where parallel lane lines are located, but it is easily affected by lane lines and causes inaccurate results; the perspective transformation matrix is ​​obtained by the vanishing lines obtained from the lane lines in the two frames of images, thereby obtaining the corresponding enhanced image data, but it does not perform additional processing on the features of the two frames of images, nor does it perform smoothing operations such as filtering on the obtained vanishing lines to eliminate the randomness of the vanishing line results. In summary, the vanishing line detection in the related technology has the problem of low accuracy.

[0005] Currently, there is no effective solution to the problem of low accuracy in vanishing line detection in related technologies. Summary of the invention

[0006] The purpose of the present application is to provide a vanishing line detection method, apparatus, computer device and computer-readable storage medium to address the deficiencies in the prior art, so as to at least solve the problem of low accuracy in vanishing line detection in the related art.

[0007] To achieve the above purpose, the technical solution adopted by this application is:

[0008] In a first aspect, an embodiment of the present application provides a vanishing line detection method, comprising:

[0009] Acquire a previous frame image and a current frame image, and perform 2D feature extraction on the previous frame image and the current frame image respectively to obtain a previous frame 2D feature and a current frame 2D feature;

[0010] Performing feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain a previous frame BEV feature and a current frame BEV feature;

[0011] Fusing the BEV feature of the previous frame with the BEV feature of the current frame to obtain a fused BEV feature;

[0012] The vanishing line height value and the vanishing line height change value are obtained according to the fused BEV features.

[0013] In some embodiments, after obtaining the vanishing line height value and the vanishing line height change value according to the fused BEV feature, the method further includes:

[0014] The vanishing line height value and the vanishing line height change value are input into a Kalman filter to obtain the pitch angle of the vehicle system.

[0015] In some embodiments, the performing feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain the previous frame BEV feature and the current frame BEV feature includes:

[0016] Get the camera parameters of the vehicle's single front-view camera;

[0017] The previous frame 2D features and the current frame 2D features are converted into the previous frame BEV features and the current frame BEV features respectively using the camera parameters.

[0018] In some embodiments, the step of fusing the previous frame BEV feature and the current frame BEV feature to obtain the fused BEV feature includes:

[0019] Obtaining an offset of a target position feature in the current frame BEV feature relative to the target position feature in the previous frame BEV feature;

[0020] Adjusting the previous frame BEV feature according to the offset to obtain an adjusted previous frame BEV feature;

[0021] The adjusted previous frame BEV feature and the current frame BEV feature are fused to obtain the fused BEV feature.

[0022] In a second aspect, an embodiment of the present application provides a vanishing line detection device, including:

[0023] A first acquisition unit is used to acquire a previous frame image and a current frame image, and perform 2D feature extraction on the previous frame image and the current frame image respectively to obtain a previous frame 2D feature and a current frame 2D feature;

[0024] A conversion unit, used to perform feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain a previous frame BEV feature and a current frame BEV feature;

[0025] A fusion unit, used for fusing the BEV feature of the previous frame with the BEV feature of the current frame to obtain a fused BEV feature;

[0026] The second acquisition unit is used to acquire the vanishing line height value and the vanishing line height change value according to the fused BEV feature.

[0027] In some embodiments, it also includes:

[0028] The third acquisition unit is used to input the vanishing line height value and the vanishing line height change value into a Kalman filter after acquiring the vanishing line height value and the vanishing line height change value according to the fused BEV feature, so as to acquire the pitch angle of the vehicle system.

[0029] In some embodiments, the conversion unit includes:

[0030] A first acquisition module, used to acquire camera parameters of a single front-view camera of a vehicle;

[0031] A conversion module is used to convert the previous frame 2D features and the current frame 2D features into the previous frame BEV features and the current frame BEV features respectively by using the camera parameters.

[0032] In some embodiments, the fusion unit comprises:

[0033] A second acquisition module, used for acquiring an offset of a target position feature in the current frame BEV feature relative to the target position feature in the previous frame BEV feature;

[0034] An adjustment module, configured to adjust the BEV feature of the previous frame according to the offset to obtain an adjusted BEV feature of the previous frame;

[0035] The fusion module is used to fuse the adjusted previous frame BEV feature and the current frame BEV feature to obtain the fused BEV feature.

[0036] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the disappearing line detection method described in the first aspect above is implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vanishing line detection method as described in the first aspect above.

[0038] The present application adopts the above technical scheme. Compared with the prior art, the vanishing line detection method provided in the embodiment of the present application obtains the previous frame image and the current frame image, and performs 2D feature extraction on the previous frame image and the current frame image respectively to obtain the previous frame 2D feature and the current frame 2D feature; performs feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain the previous frame BEV feature and the current frame BEV feature; performs feature fusion on the previous frame BEV feature and the current frame BEV feature to obtain the fused BEV feature; obtains the vanishing line height value and the vanishing line height change value according to the fused BEV feature, thereby solving the problem of low accuracy of vanishing line detection in the related art and achieving the technical effect of improving the accuracy of vanishing line detection.

[0039] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 is a structural block diagram of a mobile terminal according to an embodiment of the present application;

[0042] Figure 2 is a flow chart of a vanishing line detection method according to an embodiment of the present application;

[0043] Figure 3 It is a flow chart of a method for vanishing line detection and pitch angle filtering based on time series features according to a preferred embodiment of the present application;

[0044] Figure 4 is a structural block diagram of a disappearing line detection device according to an embodiment of the present application;

[0045] Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0047] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0048] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] This embodiment provides a mobile terminal. Figure 1 is a structural block diagram of a mobile terminal according to an embodiment of the present application. Figure 1 As shown, the mobile terminal includes: a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a wireless fidelity (WiFi) module 170, a processor 180, and a power supply 190. Those skilled in the art will understand that Figure 1 The structure of the mobile terminal shown in the figure does not constitute a limitation on the mobile terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] Combine the following Figure 1 The following is a detailed introduction to the various components of the mobile terminal:

[0052] The RF circuit 110 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 180 for processing; in addition, the designed uplink data is sent to the base station. Usually, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, referred to as LNA), a duplexer, etc. In addition, the RF circuit 110 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0053] The memory 120 can be used to store software programs and modules. The processor 180 executes various functional applications and data processing of the mobile terminal by running the software programs and modules stored in the memory 120. The memory 120 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile terminal (such as audio data, a phone book, etc.), etc. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0054] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the mobile terminal. Specifically, the input unit 130 may include a touch panel 131 and other input devices 132. The touch panel 131, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch panel 131), and drive corresponding connection devices according to a pre-set program. Optionally, the touch panel 131 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch panel 131 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 131, the input unit 130 may further include other input devices 132. Specifically, other input devices 132 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0055] The display unit 140 can be used to display information input by the user or information provided to the user and various menus of the mobile terminal. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 131 may cover the display panel 141. When the touch panel 131 detects a touch operation on or near it, it is transmitted to the processor 180 to determine the type of touch event. Subsequently, the processor 180 provides a corresponding visual output on the display panel 141 according to the type of touch event. Although in Figure 1 In the embodiment, the touch panel 131 and the display panel 141 are used as two independent components to implement the input and output functions of the mobile terminal, but in some embodiments, the touch panel 131 and the display panel 141 can be integrated to implement the input and output functions of the mobile terminal.

[0056] The mobile terminal may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 and / or the backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile terminal, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0057] The speaker 161 and microphone 162 in the audio circuit 160 can provide an audio interface between the user and the mobile terminal. The audio circuit 160 can transmit the received audio data to the speaker 161 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then the audio data is output to the processor 180 for processing, and then sent to another mobile terminal through the RF circuit 110, or the audio data is output to the memory 120 for further processing.

[0058] WiFi is a short-range wireless transmission technology. Mobile terminals can help users send and receive emails, browse web pages, and access streaming media through WiFi module 170. It provides users with wireless broadband Internet access. Figure 1 A WiFi module 170 is shown, but it is understandable that it is not an essential component of the mobile terminal and can be omitted as needed without changing the essence of the invention, or replaced with other short-range wireless transmission modules, such as a Zigbee module or a WAPI module.

[0059] The processor 180 is the control center of the mobile terminal. It uses various interfaces and lines to connect various parts of the entire mobile terminal. It executes various functions of the mobile terminal and processes data by running or executing software programs and / or modules stored in the memory 120, and calling data stored in the memory 120, so as to monitor the mobile terminal as a whole. Optionally, the processor 180 may include one or more processing units; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 180.

[0060] The mobile terminal also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 180 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0061] Although not shown, the mobile terminal may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0062] In this embodiment, the processor 180 is configured to:

[0063] Acquire a previous frame image and a current frame image, and perform 2D feature extraction on the previous frame image and the current frame image respectively to obtain a previous frame 2D feature and a current frame 2D feature;

[0064] Performing feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain a previous frame BEV feature and a current frame BEV feature;

[0065] Fusing the BEV feature of the previous frame with the BEV feature of the current frame to obtain a fused BEV feature;

[0066] The vanishing line height value and the vanishing line height change value are obtained according to the fused BEV features.

[0067] In some embodiments, the processor 180 is further configured to:

[0068] The vanishing line height value and the vanishing line height change value are input into a Kalman filter to obtain the pitch angle of the vehicle system.

[0069] In some embodiments, the processor 180 is further configured to:

[0070] Get the camera parameters of the vehicle's single front-view camera;

[0071] The previous frame 2D features and the current frame 2D features are converted into the previous frame BEV features and the current frame BEV features respectively using the camera parameters.

[0072] In some embodiments, the processor 180 is further configured to:

[0073] Obtaining an offset of a target position feature in the current frame BEV feature relative to the target position feature in the previous frame BEV feature;

[0074] Adjusting the previous frame BEV feature according to the offset to obtain an adjusted previous frame BEV feature;

[0075] The adjusted previous frame BEV feature and the current frame BEV feature are fused to obtain the fused BEV feature.

[0076] This embodiment provides a vanishing line detection method. Figure 2 is a flow chart of a disappearing line detection method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0077] Step S201, obtaining a previous frame image and a current frame image, and performing 2D feature extraction on the previous frame image and the current frame image respectively to obtain a previous frame 2D feature and a current frame 2D feature;

[0078] Step S202, performing feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain a previous frame BEV feature and a current frame BEV feature;

[0079] Step S203, fusing the BEV feature of the previous frame and the BEV feature of the current frame to obtain a fused BEV feature;

[0080] Step S204, obtaining a vanishing line height value and a vanishing line height change value according to the fused BEV feature.

[0081] The above steps provide a vanishing line detection method based on temporal feature fusion, which can effectively reduce the impact of image occlusion, vehicle motion and other factors on the vanishing line detection results, and can output the vanishing line height and the vanishing line height change value at the same time. The embodiment of the present application can effectively avoid the problem of front view feature misalignment in 2D space by performing temporal feature fusion in BEV feature space, thereby achieving the effect of improving the accuracy of vanishing line detection.

[0082] In some embodiments, the step S202 performs feature conversion on the previous frame 2D feature and the current frame 2D feature to obtain the previous frame BEV feature and the current frame BEV feature, which may include:

[0083] Get the camera parameters of the vehicle's single front-view camera;

[0084] The previous frame 2D features and the current frame 2D features are converted into the previous frame BEV features and the current frame BEV features respectively using the camera parameters.

[0085] It should be noted that the camera parameters are used in feature conversion. It can directly use the results of the previous frame or the camera parameters of the current frame obtained from the vehicle system, but they will be different from the final predicted vanishing line height. Regardless of which result is used, the final trained model will have the ability to resist disturbances to the camera's internal and external parameters during the test phase, which helps to enhance the accuracy of feature conversion and thus achieve the purpose of improving the accuracy of vanishing line detection.

[0086] In some embodiments, the step S203 performs feature fusion of the previous frame BEV feature and the current frame BEV feature to obtain the fused BEV feature, which may include:

[0087] Obtaining an offset of a target position feature in the current frame BEV feature relative to the target position feature in the previous frame BEV feature;

[0088] Adjusting the previous frame BEV feature according to the offset to obtain an adjusted previous frame BEV feature;

[0089] The adjusted previous frame BEV feature and the current frame BEV feature are fused to obtain the fused BEV feature.

[0090] It should be noted that if 2D features are directly fused, since the single front-view camera is a perspective image, the features between the two frames of images cannot be directly matched and superimposed due to factors such as the movement of the vehicle. Therefore, it is necessary to convert to the BEV space for corresponding feature fusion to ensure that the corresponding position features conform to the description of the same target. Therefore, the two BEV features can be directly superimposed, and the superposition here can be addition or splicing. The embodiment of the present application effectively avoids the problem of misalignment of front view features in 2D space by performing temporal feature fusion in the BEV feature space, thereby achieving the effect of improving the accuracy of vanishing line detection.

[0091] In some embodiments, after obtaining the vanishing line height value and the vanishing line height change value according to the fused BEV feature in step S204, the following method may be further included:

[0092] The vanishing line height value and the vanishing line height change value are input into a Kalman filter to obtain the pitch angle of the vehicle system.

[0093] The embodiment of the present application can input the vanishing line height value and the vanishing line height change value into the Kalman filter at the same time, which can effectively improve the accuracy and stability of the pitch angle prediction results, and then determine the actual posture of the vehicle according to the vanishing line detection results and the pitch angle prediction results, which plays an important role in improving the accuracy of speed and distance measurement.

[0094] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0095] When this preferred embodiment uses a single forward-looking camera to detect vanishing lines, the method based on time series features can reduce the impact of occluded targets, and by inputting the absolute value and change value of the predicted vanishing line into the filtering system, a more robust pitch angle result is obtained, thereby improving the effectiveness and stability of the detection results.

[0096] Figure 3 is a flow chart of a method for vanishing line detection and pitch angle filtering based on time series features according to a preferred embodiment of the present application, such as Figure 3 As shown, the execution process of the method includes:

[0097] 1. Get the previous frame image input_pre and the current frame image input_T, input the two frames of images into the CNN_Feature module for 2D feature extraction, and obtain the corresponding 2D features, CNN_Feature_pre and CNN_Feature_T.

[0098] 2. Input the obtained 2D features into the 2DtoBEV module, and use IPM (inverse perspective transform), LSS (Lift, Splat, Shoot), Transformer and other methods to obtain the corresponding BEV features, BEV_Feature_pre and BEV_Feature_T. It should be noted here that the camera parameter Cam_Param will be used in feature conversion. It can directly use the result of the previous frame, or it can be the current frame camera parameter obtained from the vehicle system, but it will be different from our final predicted vanishing line height VP_h. Regardless of which result is used, for the final trained model, it will have the ability to resist disturbances to the internal and external parameters of the camera during the test phase, which will help enhance the accuracy of feature transformation.

[0099] The 2DtoBEV process is briefly described in a fully connected network MLP manner, but this application does not involve the protection of such operations, as long as the 2DtoBEV method can be completed.

[0100] Generally, CNN outputs 2D features F_2D, with feature size [B, C, H, W]. F_2D is reshaped to get F_2D_tmp, with size [B, C*H*W]. After MLP with hidden_dims of BEV_h*BEV_w*BEV_c and reshap, the BEV feature map F_BEV is obtained with size [B, BEV_h, BEV_w, BEV_c]. The BEV feature size can be understood as follows. Assuming that the BEV viewing range is defined as 30m in front and back and 15m in left and right with the vehicle as the center, then each grid represents the distance of 60 / BEV_h and 30 / BEV_w in front and back. The feature dimension that each grid can represent is BEV_c.

[0101] 3. The obtained BEV features are fused, and the previous frame of BEV features are superimposed on the current BEV feature map through vehicle motion information such as rotation and translation. In general, 2D information is directly fused, but since the single-frame forward-looking camera is a perspective image, the features between the two frames of images cannot be directly matched and superimposed due to the movement of the vehicle, etc., so it is necessary to convert to the BEV space for corresponding feature fusion. This ensures that the corresponding position features meet the description of the same target.

[0102] It should be noted that, assuming that in the historical frame captured by the camera at time T-1, there is a set of zebra crossings 10 meters in front, then the corresponding position in the BEV feature map can be considered to have a corresponding zebra crossing feature. Assuming that the vehicle is traveling in a straight line and has moved forward 2 meters at time T, the distance from the zebra crossing in the current frame becomes 8 meters, then the same zebra crossing BEV feature in the current frame is also offset by 2 meters relative to the BEV feature in the historical frame. At this time, the BEV feature in the historical frame needs to be translated downward by two meters, and the remaining positions are filled with 0, so that the zebra crossing BEV feature in the historical frame corresponds to the zebra crossing BEV feature in the current frame in spatial position, so the two BEV features can be directly superimposed, either by addition or splicing.

[0103] 4. Based on the obtained BEV features, the input parameters required by the Kalman filter, namely the vanishing line height value VP_h and the vanishing line height change value VP_Δh, are obtained by outputting the regression head.

[0104] It should be noted that the output regression head is divided into two parts, one for h regression and the other for Δh regression. Analogous to the regression w and h of the bounding box in target detection, the MSE loss function is used to calculate the loss of GT and output value.

[0105] 5. Input the output height value and height change value into the Kalman filter to obtain the final pitch angle θ result, including θ and Δθ.

[0106] It should be noted that the pitch angle can be calculated from the vanishing line and the corresponding camera parameters, but the pitch angle is easily affected by vehicle movements such as road bumps, so its stability needs to be guaranteed and certain tracking technology is required. The Kalman filter is used for state estimation in dynamic multi-changing systems. Therefore, it is in line with physical laws to use the Kalman filter for the vanishing line height and the vanishing line height change as input and the pitch angle as output. An accurate pitch angle helps calculate the actual posture of the vehicle and plays an important role in improving the accuracy of speed and distance measurement.

[0107] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0108] This embodiment provides a disappearing line detection device, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, it is also possible and conceivable to implement it in hardware, or in a combination of software and hardware.

[0109] Figure 4 is a structural block diagram of a disappearing line detection device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0110] The first acquisition unit 41 is used to acquire a previous frame image and a current frame image, and perform 2D feature extraction on the previous frame image and the current frame image respectively to obtain a previous frame 2D feature and a current frame 2D feature;

[0111] A conversion unit 42, configured to perform feature conversion on the previous frame 2D feature and the current frame 2D feature to obtain a previous frame BEV feature and a current frame BEV feature;

[0112] A fusion unit 43 is used to fuse the BEV feature of the previous frame and the BEV feature of the current frame to obtain a fused BEV feature;

[0113] The second acquisition unit 44 is used to acquire the vanishing line height value and the vanishing line height change value according to the fused BEV feature.

[0114] In some embodiments, the device further comprises:

[0115] The third acquisition unit is used to input the vanishing line height value and the vanishing line height change value into a Kalman filter after acquiring the vanishing line height value and the vanishing line height change value according to the fused BEV feature, so as to acquire the pitch angle of the vehicle system.

[0116] In some embodiments, the conversion unit 42 includes:

[0117] A first acquisition module, used to acquire camera parameters of a single front-view camera of a vehicle;

[0118] A conversion module is used to convert the previous frame 2D features and the current frame 2D features into the previous frame BEV features and the current frame BEV features respectively by using the camera parameters.

[0119] In some embodiments, the fusion unit 43 includes:

[0120] A second acquisition module, used for acquiring an offset of a target position feature in the current frame BEV feature relative to the target position feature in the previous frame BEV feature;

[0121] An adjustment module, configured to adjust the BEV feature of the previous frame according to the offset to obtain an adjusted BEV feature of the previous frame;

[0122] The fusion module is used to fuse the adjusted previous frame BEV feature and the current frame BEV feature to obtain the fused BEV feature.

[0123] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0124] The embodiment provides a computer device. The vanishing line detection method according to the embodiment of the present application can be implemented by the computer device. Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0125] The computer device may include a processor 51 and a memory 52 storing computer program instructions.

[0126] Specifically, the processor 51 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0127] Among them, the memory 52 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 52 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 52 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 52 may be inside or outside the data processing device. In a specific embodiment, the memory 52 is a non-volatile memory. In a specific embodiment, the memory 52 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0128] The memory 52 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 51 .

[0129] The processor 51 reads and executes computer program instructions stored in the memory 52 to implement any one of the disappearing line detection methods in the above embodiments.

[0130] In some of the embodiments, the computer device may further include a communication interface 53 and a bus 50. Figure 5 As shown, the processor 51, the memory 52, and the communication interface 53 are connected via a bus 50 and communicate with each other.

[0131] The communication interface 53 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 53 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0132] The bus 50 includes hardware, software or both, and couples the components of the computer device to each other. The bus 50 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 50 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 50 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0133] In addition, in combination with the vanishing line detection method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any vanishing line detection method in the above embodiment is implemented.

[0134] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A vanishing line detection method, characterized in that: include: Acquire a previous frame image and a current frame image, and perform 2D feature extraction on the previous frame image and the current frame image respectively to obtain a previous frame 2D feature and a current frame 2D feature; Performing feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain a previous frame BEV feature and a current frame BEV feature; Fusing the BEV feature of the previous frame with the BEV feature of the current frame to obtain a fused BEV feature; Acquire a vanishing line height value and a vanishing line height change value according to the fused BEV features; The step of fusing the BEV feature of the previous frame with the BEV feature of the current frame to obtain the fused BEV feature comprises: obtaining the offset of the target position feature in the BEV feature of the current frame relative to the target position feature in the BEV feature of the previous frame; adjusting the BEV feature of the previous frame according to the offset to obtain the adjusted BEV feature of the previous frame; and fusing the adjusted BEV feature of the previous frame with the BEV feature of the current frame to obtain the fused BEV feature. The obtaining of the vanishing line height value and the vanishing line height change value according to the fused BEV features includes: obtaining the vanishing line height value and the vanishing line height change value by outputting a regression head based on the obtained BEV features.

2. The method according to claim 1, characterized in that After obtaining the vanishing line height value and the vanishing line height change value according to the fused BEV feature, the method further includes: The vanishing line height value and the vanishing line height change value are input into a Kalman filter to obtain the pitch angle of the vehicle system.

3. The method according to claim 1, characterized in that The performing feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain the previous frame BEV feature and the current frame BEV feature includes: Get the camera parameters of the vehicle's single front-view camera; The previous frame 2D features and the current frame 2D features are converted into the previous frame BEV features and the current frame BEV features respectively using the camera parameters.

4. A vanishing line detection device, characterized in that: include: A first acquisition unit is used to acquire a previous frame image and a current frame image, and perform 2D feature extraction on the previous frame image and the current frame image respectively to obtain a previous frame 2D feature and a current frame 2D feature; A conversion unit, used to perform feature conversion on the previous frame 2D feature and the current frame 2D feature respectively to obtain a previous frame BEV feature and a current frame BEV feature; A fusion unit, used for fusing the BEV feature of the previous frame with the BEV feature of the current frame to obtain a fused BEV feature; A second acquisition unit, configured to acquire a vanishing line height value and a vanishing line height change value according to the fused BEV feature; The fusion unit includes: a second acquisition module, used to obtain the offset of the target position feature in the current frame BEV feature relative to the target position feature in the previous frame BEV feature; an adjustment module, used to adjust the previous frame BEV feature according to the offset to obtain the adjusted previous frame BEV feature; a fusion module, used to perform feature fusion on the adjusted previous frame BEV feature and the current frame BEV feature to obtain the fused BEV feature; The second acquisition unit is used to obtain the vanishing line height value and the vanishing line height change value by outputting a regression head based on the obtained BEV feature.

5. The device according to claim 4, characterized in that Also includes: The third acquisition unit is used to input the vanishing line height value and the vanishing line height change value into a Kalman filter after acquiring the vanishing line height value and the vanishing line height change value according to the fused BEV feature, so as to acquire the pitch angle of the vehicle system.

6. The device according to claim 4, characterized in that The conversion unit comprises: A first acquisition module, used to acquire camera parameters of a single front-view camera of a vehicle; A conversion module is used to convert the previous frame 2D features and the current frame 2D features into the previous frame BEV features and the current frame BEV features respectively by using the camera parameters.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images