Vehicle continuous lane change recognition method, device, equipment and storage medium

Abstract

The present disclosure provides a method, device, equipment and storage medium for identifying continuous lane changes of vehicles, wherein the method comprises: obtaining a video stream to be tested; wherein the video stream to be tested includes at least two lanes; based on vehicle detection and tracking detection of the video stream to be tested, determining the vehicle in the video stream to be tested and the lane where the vehicle is located; determining the entry time of the vehicle from the original driving lane into the adjacent lane and the exit time of the vehicle leaving the adjacent lane; based on the lane change time difference of the vehicle being less than a preset time difference threshold, determining that the vehicle has changed lanes continuously; wherein the lane change time difference is the difference between the exit time and the entry time.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for identifying continuous lane changes of a vehicle. Background Art

[0002] At present, vehicles are still important participants in the field of transportation, and vehicle driving behavior is also a hot topic in the field of transportation. In real life, some vehicles have the behavior of changing lanes continuously, which seriously affects or endangers the normal traffic order. However, methods such as on-site law enforcement or remote manual inspections have problems such as low recognition efficiency and small coverage. There is an urgent need for a video-based vehicle continuous lane change recognition method to assist in identifying illegal vehicles with continuous lane change behavior. Summary of the invention

[0003] The present disclosure provides a method, device, equipment and storage medium for identifying continuous lane changes of a vehicle.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for identifying continuous lane changes of vehicles is provided, the method comprising: obtaining a video stream to be tested; wherein the video stream to be tested includes at least two lanes; based on vehicle detection and tracking detection of the video stream to be tested, determining the vehicle in the video stream to be tested and the lane where the vehicle is located; determining an entry time of the vehicle from the original driving lane into an adjacent lane and an exit time of the vehicle leaving the adjacent lane; determining that the vehicle has changed lanes continuously based on the lane change time difference of the vehicle being less than a preset time difference threshold; wherein the lane change time difference is the difference between the exit time and the entry time.

[0005] In some optional embodiments, determining the vehicle in the video stream to be tested and the lane where the vehicle is located based on vehicle detection and tracking detection of the video stream to be tested includes: performing vehicle detection and tracking detection on multiple frames of images included in the video stream to be tested to obtain multiple frames of vehicle foreground images of the vehicle; and determining the lane where the vehicle is located in each frame of the image based on the relative position of the vehicle and the lane in each frame of the vehicle foreground image.

[0006] In some optional embodiments, determining the lane where the vehicle is located in each frame of the image based on the relative position of the vehicle and the lane in the vehicle foreground image of each frame includes: detecting the vehicle foreground image to determine a vehicle area of ​​the vehicle in the vehicle foreground image; wherein the vehicle area is used to represent the road surface area occupied by the vehicle; and determining the lane where the vehicle is located based on the relative position of the vehicle area and the lane.

[0007] In some optional embodiments, detecting the vehicle foreground image and determining the vehicle area of ​​the vehicle in the vehicle foreground image includes: inputting the vehicle foreground image into a pre-trained key point detection network, and having the key point detection network detect wheel key points of the vehicle in the image; and determining the vehicle area of ​​the vehicle in the image based on a polygon enclosed by the wheel key points.

[0008] In some optional embodiments, determining the lane where the vehicle is located based on the relative position of the vehicle area and the lane includes: determining the overlapping area between the vehicle area and the lane as the first area; determining the proportion of the first area to the vehicle area as the vehicle overlap degree; and when the vehicle overlap degree is greater than a preset overlap degree threshold, determining the lane corresponding to the vehicle overlap degree as the lane where the vehicle is located.

[0009] In some optional embodiments, determining the entry time of the vehicle from the original driving lane to the adjacent lane and the exit time of the vehicle leaving the adjacent lane includes: determining the time corresponding to the image frame in the video stream to be tested when the lane where the vehicle is located changes from the original driving lane to the adjacent lane as the entry time; and determining the time corresponding to the image frame in the video stream to be tested when the vehicle leaves the adjacent lane as the exit time.

[0010] In some optional embodiments, after determining that the vehicle changes lanes continuously, the method further includes: initiating an alarm and / or saving an image frame of the vehicle changing lanes.

[0011] According to a second aspect of an embodiment of the present disclosure, a vehicle continuous lane change recognition device is provided, the device comprising: a video stream acquisition module, used to acquire a video stream to be tested; wherein the video stream to be tested includes at least two lanes; a vehicle lane determination module, used to determine the vehicle in the video stream to be tested and the lane where the vehicle is located based on vehicle detection and tracking detection of the video stream to be tested; a time determination module, used to determine the entry time of the vehicle from the original driving lane into the adjacent lane and the exit time of the vehicle leaving the adjacent lane; a continuous lane change determination module, used to determine that the vehicle has changed lanes continuously based on the time difference of the lane change of the vehicle being less than a preset time difference threshold; wherein the lane change time difference is the difference between the exit time and the entry time.

[0012] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the vehicle continuous lane change recognition method described in any one of the first aspects when executing the program.

[0013] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the vehicle continuous lane change recognition method described in any one of the first aspects is implemented.

[0014] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, wherein when the program is executed by a processor, the vehicle continuous lane change recognition method described in any one of the first aspects is implemented.

[0015] In the disclosed embodiment, the two lane change times of the vehicle entering and leaving the adjacent lane are determined, and the driving time of the vehicle in the adjacent lane is determined based on the two lane change times, that is, the lane change time difference between the two consecutive lane change times of the vehicle is determined, and if the lane change time difference is too small, the vehicle is determined to have changed lanes continuously. This method can identify the continuous lane change of the vehicle during the driving process based on the video stream, and can automatically detect without relying on manual methods, thereby improving the efficiency of the continuous lane change identification of the vehicle and helping to improve traffic violations.

[0016] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] Figure 1 is a flow chart of a method for identifying a vehicle continuous lane change according to an exemplary embodiment;

[0019] Figure 2 is a schematic diagram of a vehicle changing lanes according to an exemplary embodiment;

[0020] Figure 3 is a flow chart of a method for determining a time according to an exemplary embodiment;

[0021] Figure 4 is a flow chart of a lane determination method according to an exemplary embodiment;

[0022] Figure 5 is a schematic diagram of an image in a video stream to be tested according to an exemplary embodiment;

[0023] Figure 6 is a schematic diagram of a vehicle foreground image according to an exemplary embodiment;

[0024] Figure 7 is a flow chart of another lane determination method according to an exemplary embodiment;

[0025] Figure 8 is a flow chart of a vehicle area determination method according to an exemplary embodiment;

[0026] Fig. 9 is a schematic diagram of a network structure of a key point detection network according to an exemplary embodiment;

[0027] Fig.10 is a flow chart of another lane determination method according to an exemplary embodiment;

[0028] Fig.11 is a schematic diagram of a vehicle continuous lane change recognition device according to an exemplary embodiment;

[0029] Fig.12 is a schematic diagram of another vehicle continuous lane change recognition device according to an exemplary embodiment;

[0030] Fig.13 is a schematic diagram of a lane determination submodule according to an exemplary embodiment;

[0031] Fig.14 is a schematic diagram of another vehicle continuous lane change recognition device according to an exemplary embodiment;

[0032] Fig.15 The diagram is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The specific methods described in the following exemplary embodiments do not represent all schemes consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0036] In order to make the vehicle continuous lane change recognition method provided by the present disclosure clearer, the execution process of the solution provided by the present disclosure is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for identifying a vehicle continuous lane change according to an embodiment of the present disclosure. Figure 1 As shown, the process includes:

[0038] Step 101, obtaining a video stream to be tested; wherein the video stream to be tested includes at least two lanes.

[0039] In the disclosed embodiment, the video stream for which continuous lane change identification of a vehicle is required may be referred to as a video stream to be tested. Furthermore, the video stream to be tested includes at least two lanes in the picture. The two lanes included in the video stream picture may be driven by a vehicle, or the vehicle may change lanes between different lanes.

[0040] There are many specific ways to obtain the video stream to be tested, which are not limited in this embodiment. For example, taking the identification of continuous lane changes of vehicles on a highway as an example, in this step, the video acquisition equipment installed on the highway can be reused to obtain the video stream to be tested. In this way, the original video acquisition equipment can be reused, reducing the hardware cost.

[0041] Step 102: Based on vehicle detection and tracking detection of the video stream to be tested, determine the vehicle in the video stream to be tested and the lane where the vehicle is located.

[0042] This step can detect the video stream to be tested, and determine the vehicles included in the video stream to be tested and the lanes where the vehicles are located. Among them, the detection of the video stream to be tested may include vehicle detection and tracking detection. Through the vehicle detection of the video stream to be tested, the vehicles included in the video stream to be tested can be identified, and a plurality of vehicle detection frames can be obtained accordingly. It can be understood that the specific manner of performing vehicle detection on the video stream to be tested is not limited by the embodiments of the present disclosure. For example, a network model that can perform vehicle detection in the related technology can be reused to achieve vehicle detection of vehicles in the video stream to be tested. Alternatively, a detection network that meets the detection requirements can be obtained based on the collected sample data and trained based on a deep neural network model, and the trained detection network can be used to achieve vehicle detection of vehicles in the video stream to be tested.

[0043] After vehicle detection is performed on the video stream to be tested, different image frames included in the video stream to be tested may include multiple different vehicles. However, the same vehicle between different image frames is not associated, that is, the same vehicle in different image frames cannot be identified. Therefore, on the basis of vehicle detection on the video stream to be tested, this step also includes tracking detection on the video stream to be tested. Through tracking detection, the same vehicles appearing in the video stream to be tested can be associated, that is, the vehicle tracking sequence image of each vehicle can be obtained in combination with the vehicle detection frame obtained previously. For example, the same vehicle included in different image frames can be identified, and the same vehicle in the video stream to be tested can be identified. It can be understood that the specific method of tracking detection on the video stream to be tested can include multiple, and the embodiment of the present disclosure is not limited. Similar to vehicle detection on the video stream to be tested, the specific method of tracking detection on the video stream to be tested can be based on the tracking algorithm in the relevant technology to achieve tracking detection of the video stream to be tested; or it can be based on the deep neural network to train a detection model that meets the requirements, so that the detection model can be used to achieve tracking detection of the video stream to be tested.

[0044] On the basis of vehicle detection and tracking detection in the video stream to be tested, after determining the vehicle in the video stream to be tested, this step can further determine the lane the vehicle is in. The detailed process of determining the lane where the vehicle is located based on the video stream to be tested will be described in detail later and will not be repeated here.

[0045] Step 103, determining the entry time of the vehicle from the original driving lane into the adjacent lane and the exit time of the vehicle leaving the adjacent lane.

[0046] After determining the vehicle in the video stream to be tested and the lane where the vehicle is located, this step can further determine the entry time of the vehicle from the original driving lane to the adjacent lane, and determine the exit time of the vehicle leaving the adjacent lane.

[0047] The lane that the vehicle enters after leaving the adjacent lane may be the original lane that the vehicle has traveled before; or the lane that the vehicle enters after leaving the adjacent lane may be another lane other than the original lane. That is, the lane that the vehicle is in before entering the adjacent lane and the lane that the vehicle enters after leaving the adjacent lane may be the same lane or different lanes.

[0048] Figure 2 A schematic diagram of a vehicle changing lanes during driving is shown. Figure 2 There are three lanes: L1, L2 and L3; when the vehicle is in the Car1 position, it is changing lanes from lane L1 to lane L2, and when the vehicle is in the Car2 position, it is changing lanes from lane L2 to lane L3, so as to achieve the change from lane L1 to lane L3. Figure 2 For example, in this step, lane L1 can be used as the original driving lane, lane L2 can be used as the adjacent lane, the moment when the vehicle enters lane L2 from lane L1 can be used as the entry time; and the moment when the vehicle leaves lane L2 and enters lane L3 can be used as the exit time.

[0049] The lane that the vehicle was in before entering the adjacent lane and the lane that the vehicle entered after leaving the adjacent lane may be different lanes. For example, the moment when the vehicle enters lane L2 from lane L1 may be regarded as the entry time, and the moment when the vehicle enters lane L3 from lane L2 may be regarded as the exit time.

[0050] The lane that the vehicle was in before entering the adjacent lane and the lane that the vehicle entered after leaving the adjacent lane can be the same lane. For example, the moment when the vehicle enters lane L2 from lane L1 can be regarded as the entry time, and the moment when the vehicle re-enters lane L1 from lane L2 can be regarded as the exit time.

[0051] It should be noted that whether the lane in which the vehicle enters an adjacent lane and leaves an adjacent lane is the same lane can be flexibly specified or set according to specific scenario requirements, and this embodiment does not specifically limit it.

[0052] In some optional embodiments, the specific implementation of step 103 is as follows: Figure 3 As shown, the following steps may be included:

[0053] Step 301, determining the time corresponding to the image frame in which the lane where the vehicle is located in the video stream to be tested changes from the original driving lane to the adjacent lane as the entry time;

[0054] Step 302: Determine the time corresponding to the image frame in the video stream to be tested when the vehicle leaves the adjacent lane as the leaving time.

[0055] In the disclosed embodiment, when tracking and detecting a vehicle, the current time information of each image frame can be obtained, that is, the entry and exit times of the vehicle can be determined based on the time corresponding to the image frame in the video stream to be tested, combined with the change of the lane where the vehicle is located in the image frame. Specifically, the image frame in which the lane where the vehicle is located in the image frame in the video stream to be tested changes from the original driving lane to the adjacent lane can be determined as the image frame corresponding to the vehicle changing lanes from the original driving lane to the adjacent lane, and the time corresponding to the image frame is used as the entry time of the vehicle. Similarly, the image frame in which the lane where the vehicle is located in the image frame in the video stream to be tested changes from the adjacent lane to the non-adjacent lane can be determined as the image frame corresponding to the vehicle leaving the adjacent lane, and the time corresponding to the image frame is used as the departure time of the vehicle.

[0056] Step 104, determining that the vehicle changes lanes continuously based on that the lane change time difference of the vehicle is less than a preset time difference threshold; wherein the lane change time difference is the difference between the exit time and the entry time.

[0057] After determining the entry time and the exit time, this step can further determine the total duration between the two times as the driving time of the vehicle in the adjacent lane, that is, the lane change time difference between the two lane change times corresponding to the entry time and the exit time of the vehicle.

[0058] In the disclosed embodiment, a certain time difference threshold may be pre-set for comparison with the lane change time difference to determine whether the two lane change times of the vehicle are too close, that is, to determine whether the vehicle has achieved two consecutive lane change behaviors in a short period of time.

[0059] For example, a time difference threshold t=3 seconds may be preset, and if the time difference of the vehicle's lane change is less than 3 seconds, it is determined that the vehicle changes lanes continuously. Further, an alarm message may be issued or relevant information of the vehicle may be recorded for further processing.

[0060] In one possible implementation, an alarm may be initiated and / or image frames of the vehicle changing lanes may be saved. The specific method of initiating an alarm may include a variety of implementations, which are not limited by the embodiments of the present disclosure. For example, an alarm prompt message may be sent to the detection system of the traffic detection department to remind relevant staff that a vehicle has been detected to be changing lanes continuously. For another example, an alarm may be directly initiated to the device used by the driver to directly remind the driver to drive carefully. For example, an alarm message may be sent directly to the mobile phone used by the driver via text message or other means. In addition, after detecting that the vehicle has changed lanes continuously, the relevant image frames or video streams of the vehicle that has changed lanes continuously may be further saved to be used as evidence during the processing process.

[0061] In the disclosed embodiment, by determining the two lane change times when the vehicle enters and leaves the adjacent lane, the driving time of the vehicle in the adjacent lane is determined according to the two lane change times, that is, the lane change time difference between the two consecutive lane change times of the vehicle is determined, and if the lane change time difference is too small, the vehicle is determined to have changed lanes continuously. This method can identify the continuous lane change of the vehicle during the driving process based on the video stream, and can automatically detect it without relying on manual methods, thereby improving the efficiency of identifying the continuous lane change of the vehicle and helping to improve traffic violations.

[0062] In some optional embodiments, the specific implementation of determining the lane where the vehicle is located based on the video stream to be tested in step 102 is as follows: Figure 4 As shown, the following steps may be included:

[0063] Step 401 : performing vehicle detection and tracking detection on multiple frames of images included in the video stream to be tested, respectively, to obtain multiple frames of vehicle foreground images of the vehicle.

[0064] Since the video stream to be tested is composed of multiple frames of continuous images, the process of detecting the video stream to be tested is actually a process of detecting the multiple frames of images included in the video stream to be tested respectively and determining the lane where the vehicle is located in the multiple frames of images.

[0065] In the process of detecting the multiple frames of images included in the video stream to be tested, the video stream to be tested can be preprocessed in advance to reduce the number of image frames included in the video stream, thereby reducing the amount of detection calculation and improving detection efficiency. In some optional embodiments, multiple frames of images can be extracted from the video stream to be tested according to preset rules; the extracted multiple frames of images are sorted in chronological order to obtain a set of vehicle images; the multiple frames of images included in the image set are detected separately to determine the lane where the vehicle is located in the multiple frames of images.

[0066] In the above embodiment, image extraction rules can be preset, image frames can be extracted from the video stream to be tested according to the preset rules, and the extracted vehicle image set can be used as a new video stream to be tested, and the images contained therein can be detected. For example, 5 frames of images can be uniformly extracted from the video stream every 1 second, and then the images extracted in a certain period of time can be sorted according to the time sequence of the images in the video stream to be tested, to obtain a vehicle image set as a new video stream to be tested and detect the images therein.

[0067] In this method, images in the video stream to be tested can be extracted based on preset rules, which can reduce the number of image frames in the video stream to be tested to a certain extent, thereby reducing the computational complexity of detecting images in the video stream to be tested and improving the detection efficiency of the lane where the vehicle is located.

[0068] In the disclosed embodiment, the image included in the video stream to be tested is an original size image captured by an image acquisition device. Due to differences in image acquisition devices, the size of the captured image may be different. Therefore, in this step, vehicle detection and tracking detection can be performed on multiple frames of images included in the video stream to be tested, respectively, to obtain multiple frames of vehicle foreground images of preset sizes. Among them, the vehicle foreground image includes only one vehicle.

[0069] For example, the image can be cropped into a vehicle foreground image of a preset size based on the vehicle detection frame of the vehicle detected from the image. This method crops the image into a vehicle foreground image of a uniform size, which can more conveniently perform further detection on the vehicle in the vehicle foreground image. Figure 5 The image shown is cropped to obtain Figure 6 The vehicle foreground image shown. Figure 6 The vehicle foreground image shown may be an image of a preset size, and the vehicle foreground image includes only one vehicle.

[0070] Step 402: determining the lane where the vehicle is located in each frame of the image based on the relative position between the vehicle and the lane in each frame of the vehicle foreground image.

[0071] After obtaining a multi-frame foreground image of the vehicle according to the multi-frame images included in the video stream to be tested, this step can further detect the vehicle in the vehicle foreground image and determine the lane where the vehicle in the image is located.

[0072] In some optional embodiments, the specific implementation of step 402 is as follows: Figure 7 The following steps may be included:

[0073] Step 701, detecting the vehicle foreground image to determine the vehicle area of ​​the vehicle in the vehicle foreground image; wherein the vehicle area is used to represent the road area occupied by the vehicle.

[0074] When a vehicle is traveling on a road, the area of ​​the road occupied by the vehicle can best represent the actual position of the vehicle. In this step, the vehicle foreground image can be detected and the road area occupied by the vehicle in the vehicle foreground image can be used as the vehicle area of ​​the vehicle in the vehicle foreground image.

[0075] Figure 5 As shown in FIG. 1 , it is one of the multiple frames of images included in the video stream to be tested. Box1 is the vehicle detection frame determined in the relevant image detection technology, and the vehicle region determined by detecting the vehicle foreground image in this step is Region1. Figure 5As shown, the area represented by Region1 can be closer to the area of ​​the road actually occupied by the vehicle, so the vehicle area can more accurately represent the actual position of the vehicle.

[0076] Step 702: Determine the lane where the vehicle is located based on the relative position of the vehicle area and the lane.

[0077] After determining the vehicle area of ​​the vehicle in the vehicle foreground image, this step can further determine the lane where the vehicle is located based on the relative position of the vehicle area and the lane. Figure 5 The image shown includes lanes La1 and L2. Region1 is used as the vehicle region of the vehicle. In this step, it can be determined whether the vehicle is located on L2 based on the relative position between Region1 and lane L2.

[0078] The specific method of determining the lane in which the vehicle is located according to the relative position of the vehicle area and the lane can be flexibly implemented according to the specific application, and this embodiment does not limit it. For example, the lane in which the vehicle is located can be determined when the overlapping area between the vehicle area and the lane is large enough.

[0079] In the disclosed embodiment, the vehicle foreground image can be detected and the road surface area occupied by the vehicle in the vehicle foreground image can be determined as the vehicle area, so that the lane where the vehicle is located can be further determined based on the relative position of the vehicle area and the lane. In this way, since the road surface area occupied by the vehicle is detected from the vehicle foreground image as the vehicle area, the actual position of the vehicle can be more accurately represented, so that the lane where the vehicle is located can be more accurately determined. On the basis of accurately determining the lane where the vehicle is located, the continuous lane change of the vehicle can be more accurately identified.

[0080] In some optional embodiments, the specific implementation of step 701 is as follows: Figure 8 As shown, the following steps may be included:

[0081] Step 801: input the vehicle foreground image into a pre-trained key point detection network, and the key point detection network detects the wheel key points of the vehicle in the image.

[0082] In the disclosed embodiment, a key point detection network that can detect the key points of the wheels of the vehicle in the vehicle foreground image can be pre-trained. The key point detection network can be trained based on any learnable machine learning model or neural network model. In the disclosed embodiment, the specific form of the key point detection network is not limited.

[0083] For example, Fig. 9A schematic diagram of the network structure of a key point detection network is given. ResNet is used as the backbone network for extracting image features. The input of the backbone network can be a vehicle foreground image. After the convolution operation of the backbone network, the spatial resolution of the feature map gradually decreases, and the semantic features become more obvious. It is understandable that the backbone network can include more other forms of backbone networks in addition to ResNet, such as other types of general convolutional neural network structures such as GoogLeNet, VGGNet or ShuffleNet.

[0084] Furthermore, feature pyramid networks (FPN) can be used to extract multi-scale features. Specifically, the resolution of low-resolution feature maps can be restored through deconvolution and element-level addition operations. The output of FPN has a resolution equivalent to the original. Figure 4 The feature map is one-half the size of 32*32.

[0085] Furthermore, the output of FPN can be further convolved to predict five positioning heat maps. The five positioning heat maps correspond to the left front wheel, left rear wheel, right rear wheel, right front wheel and background of the vehicle. The wheel key points in the foreground image of the vehicle can be further determined based on the positioning heat maps.

[0086] In a possible implementation, the wheel key point includes a position point where the wheel directly contacts the road surface, or includes a wheel center point. The wheel key point is used to indicate the position of the wheel. It is understandable that different vehicles have different numbers of wheels, so the number of wheel key points may also vary depending on the vehicle.

[0087] For example, the position coordinates of four wheel key points can be obtained. Exemplarily, the wheel key points include: a left front wheel key point, a left rear wheel key point, a right rear wheel key point, and a right front wheel key point. Figure 6 As shown, the wheel key points of the vehicle may include a left front wheel key point S1, a left rear wheel key point S2, a right rear wheel key point S3 and a right front wheel key point S4.

[0088] Step 802: Determine a vehicle region of the vehicle in the image based on a polygon enclosed by the wheel key points.

[0089] Since the wheels of a vehicle are in direct contact with the road surface during driving, the vehicle area formed by the wheel key points can more accurately represent the road surface area occupied by the vehicle. In this step, the vehicle area of ​​the vehicle can be determined based on the wheel key points detected in the vehicle foreground image. The specific method of determining the vehicle area based on the wheel key points may include multiple implementations, which are not limited in this embodiment.

[0090] In a possible implementation, a polygonal area formed by multiple wheel key points can be used as the vehicle area. Figure 6 As shown, when the vehicle includes four wheel key points, the quadrilateral area formed by the four wheel key points can be used as the vehicle area of ​​the vehicle, that is, the quadrilateral S1S2S3S4 can be used as the vehicle area of ​​the vehicle.

[0091] In the disclosed embodiment, by detecting the wheel key points of the vehicle in the vehicle foreground image, the vehicle area of ​​the vehicle can be determined based on the wheel key points, more accurately indicating the road area occupied by the vehicle, and then the lane where the vehicle is located can be accurately determined based on the relative position of the vehicle area and the lane.

[0092] In some optional embodiments, the specific implementation of step 702 is as follows: Fig.10 As shown, the following steps may be included:

[0093] Step 1001, determining an overlapping area between the vehicle area and the lane as a first area;

[0094] Step 1002: Determine the ratio of the first area to the vehicle area as the vehicle overlap.

[0095] by Figure 5 For example, the vehicle region Region1 and the lane L2 are included, and the overlapping region R1 between the vehicle region Region1 and the lane L2 is included. The overlapping region R1 can be referred to as the first region. In this step, the ratio of the first region R1 to the vehicle region Region1 can be determined as the vehicle overlap, that is, the vehicle overlap is calculated as: R1 / Region1.

[0096] In a possible implementation, after obtaining the overlapping area between the vehicle area and the lane, i.e., determining the first area, the area of ​​the first area can be further calculated, and when the area of ​​the first area is greater than a preset area threshold, it is determined that the vehicle is located in the corresponding lane. The preset area threshold can be determined based on the total area of ​​the vehicle area. Exemplarily, the preset area threshold can be pre-defined as half of the total area of ​​the vehicle area, so the area threshold can be different depending on the total area of ​​the vehicle area.

[0097] Step 1003: When the vehicle overlap is greater than a preset overlap threshold, the lane corresponding to the vehicle overlap is determined as the lane where the vehicle is located.

[0098] The embodiment of the present disclosure may pre-set an overlap threshold as a comparison value of the vehicle overlap to determine whether the vehicle is in the corresponding lane. For example, the overlap threshold may be set to 0.5, and when the vehicle overlap R1 / Region1 is greater than 0.5, it may be determined that the vehicle is in lane L2.

[0099] In the disclosed embodiment, the ratio of the overlapping area between the vehicle area and the lane to the vehicle area can be defined as the vehicle overlap, and the vehicle overlap is compared with a preset overlap threshold, and when the vehicle overlap is greater than the overlap threshold, it is determined that the vehicle is located in the corresponding lane. In this method of determining the lane where the vehicle is located, since the vehicle area itself is the road surface area occupied by the vehicle, the ratio of the overlapping area to the vehicle area is used as the basis for judgment, and the lane where the vehicle is located can be more accurately determined. After accurately determining the lane where the vehicle is located, the lane change behavior of the vehicle can be more accurately detected, and the continuous lane change behavior of the vehicle can be more accurately identified.

[0100] Fig.11 As shown, the present disclosure provides a vehicle continuous lane change recognition device, which can execute the vehicle continuous lane change recognition method of any embodiment of the present disclosure. The device may include a video stream acquisition module 1101, a vehicle lane determination module 1102, a time determination module 1103 and a continuous lane change determination module 1104. Among them:

[0101] The video stream acquisition module 1101 is used to acquire the video stream to be tested; wherein the video stream to be tested includes at least two lanes;

[0102] A vehicle lane determination module 1102 is used to determine the vehicle in the video stream to be tested and the lane where the vehicle is located based on vehicle detection and tracking detection of the video stream to be tested;

[0103] The time determination module 1103 is used to determine the entry time of the vehicle from the original driving lane to the adjacent lane and the exit time of the vehicle leaving the adjacent lane;

[0104] The continuous lane change determination module 1104 is used to determine that the vehicle changes lanes continuously based on the lane change time difference of the vehicle being less than a preset time difference threshold; wherein the lane change time difference is the difference between the exit time and the entry time.

[0105] Optional, such as Fig.12 As shown, the vehicle lane determination module 1102 includes:

[0106] The foreground image acquisition submodule 1201 is used to perform vehicle detection and tracking detection on multiple frames of images included in the video stream to be tested, and obtain multiple frames of vehicle foreground images of the vehicle;

[0107] The lane determination submodule 1202 is used to determine the lane where the vehicle is located in each frame of the image based on the relative position between the vehicle and the lane in each frame of the vehicle foreground image.

[0108] Optional, such as Fig.13 As shown, the lane determination submodule 1202 includes:

[0109] The vehicle area determination submodule 1301 is used to detect the vehicle foreground image and determine the vehicle area of ​​the vehicle in the vehicle foreground image; wherein the vehicle area is used to represent the road area occupied by the vehicle;

[0110] The vehicle lane determination submodule 1302 is used to determine the lane where the vehicle is located according to the relative position between the vehicle area and the lane.

[0111] Optionally, the vehicle area determination submodule 1301, when used to detect the vehicle foreground image and determine the vehicle area of ​​the vehicle in the vehicle foreground image, includes: inputting the vehicle foreground image into a pre-trained key point detection network, and having the key point detection network detect the wheel key points of the vehicle in the image; and determining the vehicle area of ​​the vehicle in the image based on the polygon enclosed by the wheel key points.

[0112] Optionally, the vehicle lane determination submodule 1302, when used to determine the lane where the vehicle is located based on the relative positions of the vehicle area and the lane, includes: determining the overlapping area between the vehicle area and the lane as a first area; determining the proportion of the first area to the vehicle area as the vehicle overlap degree; and when the vehicle overlap degree is greater than a preset overlap degree threshold, determining the lane corresponding to the vehicle overlap degree as the lane where the vehicle is located.

[0113] Optionally, the moment determination module 1103 is used to determine the entry moment of the vehicle from the original driving lane to the adjacent lane and the exit moment of the vehicle leaving the adjacent lane, including: determining the moment corresponding to the image frame in the video stream to be tested when the lane where the vehicle is located changes from the original driving lane to the adjacent lane as the entry moment; determining the moment corresponding to the image frame in the video stream to be tested when the vehicle leaves the adjacent lane as the exit moment.

[0114] Optional, such as Fig.14 As shown, the device further includes: a result processing module 1401, which is used to initiate an alarm and / or save image frames of the vehicle changing lanes.

[0115] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of at least one embodiment of the present disclosure. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0116] The present disclosure also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the vehicle continuous lane change recognition method of any embodiment of the present disclosure when executing the program.

[0117] Fig.15 A more specific hardware structure diagram of a computer device provided by an embodiment of the present disclosure is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1050.

[0118] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0119] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0120] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0121] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0122] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0123] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0124] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the vehicle continuous lane change recognition method of any embodiment of the present disclosure can be implemented.

[0125] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present disclosure is not limited thereto.

[0126] In some optional embodiments, the embodiments of the present disclosure provide a computer program product, including computer readable codes, and when the computer readable codes are executed on a device, a processor in the device executes the method for realizing the continuous lane change recognition method for a vehicle provided in any of the above embodiments. The computer program product can be implemented in hardware, software, or a combination thereof.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the inventions claimed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not claimed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0129] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for identifying continuous lane changes of vehicles, characterized in that: The method comprises: Acquire a video stream to be tested; wherein the video stream to be tested includes at least two lanes; Based on vehicle detection and tracking detection of the video stream to be tested, determining the vehicle in the video stream to be tested and the lane where the vehicle is located; Determine the entry time of the vehicle from the original driving lane into the adjacent lane and the exit time of the vehicle from the adjacent lane; Based on the lane change time difference of the vehicle being less than a preset time difference threshold, determining that the vehicle changes lanes continuously; wherein the lane change time difference is the difference between the exit time and the entry time; The determining of the entry time of the vehicle from the original driving lane into the adjacent lane and the exit time of the vehicle from the adjacent lane comprises: Determine the time corresponding to the image frame at which the lane where the vehicle is located in the video stream to be tested changes from the original driving lane to the adjacent lane as the entry time; Determine the time corresponding to the image frame in the video stream to be tested that the vehicle leaves the adjacent lane as the leaving time; Wherein, when the overlap between the vehicle area and the adjacent lane changes from being less than or equal to the overlap threshold to being greater than the overlap threshold, it is determined that the lane where the vehicle is located changes from the original driving lane to the adjacent lane; When the degree of overlap between the vehicle region and the adjacent lane changes from being greater than the degree of overlap threshold to being less than the degree of overlap threshold, it is determined that the vehicle leaves the adjacent lane.

2. The method according to claim 1, characterized in that The determining of the vehicle in the video stream to be tested and the lane where the vehicle is located based on the vehicle detection and tracking detection of the video stream to be tested includes: Performing vehicle detection and tracking detection on multiple frames of images included in the video stream to be tested, respectively, to obtain multiple frames of vehicle foreground images of the vehicle; Based on the relative position between the vehicle and the lane in each frame of the vehicle foreground image, the lane where the vehicle is located in each frame of the image is determined.

3. The method according to claim 2, characterized in that The determining the lane where the vehicle is located in each frame of the image based on the relative position between the vehicle and the lane in each frame of the vehicle foreground image includes: Detecting the vehicle foreground image to determine a vehicle region of the vehicle in the vehicle foreground image; wherein the vehicle region is used to represent a road surface region occupied by the vehicle; The lane where the vehicle is located is determined according to the relative position of the vehicle area and the lane.

4. The method according to claim 3, characterized in that The detecting the vehicle foreground image to determine the vehicle region of the vehicle in the vehicle foreground image includes: Inputting the vehicle foreground image into a pre-trained key point detection network, and detecting wheel key points of the vehicle in the image by the key point detection network; A vehicle region of the vehicle in the image is determined based on a polygon enclosed by the wheel key points.

5. The method according to any one of claims 3 to 4, characterized in that The determining the lane where the vehicle is located according to the relative position of the vehicle area and the lane includes: Determine an overlapping area between the vehicle area and the lane as a first area; determining the ratio of the first area to the vehicle area as the vehicle overlap; When the vehicle overlap is greater than a preset overlap threshold, the lane corresponding to the vehicle overlap is determined as the lane where the vehicle is located.

6. The method according to claim 1, characterized in that After determining that the vehicle changes lanes continuously, the method further includes: Initiate an alarm and / or save the image frame of the vehicle changing lanes.

7. A vehicle continuous lane change recognition device, characterized in that: The device comprises: A video stream acquisition module, used to acquire a video stream to be tested; wherein the video stream to be tested includes at least two lanes; A vehicle lane determination module, used to determine the vehicle in the video stream to be tested and the lane where the vehicle is located based on vehicle detection and tracking detection of the video stream to be tested; A time determination module, used to determine the entry time of the vehicle from the original driving lane to the adjacent lane and the exit time of the vehicle leaving the adjacent lane; a continuous lane change determination module, configured to determine that the vehicle changes lanes continuously based on the lane change time difference of the vehicle being less than a preset time difference threshold; wherein the lane change time difference is the difference between the exit time and the entry time; The time determination module is specifically used to determine the time corresponding to the image frame in the video stream to be tested that the lane where the vehicle is located changes from the original driving lane to the adjacent lane as the entry time; and determine the time corresponding to the image frame in the video stream to be tested that the vehicle leaves the adjacent lane as the exit time; Wherein, when the overlap between the vehicle area and the adjacent lane changes from being less than or equal to the overlap threshold to being greater than the overlap threshold, it is determined that the lane where the vehicle is located changes from the original driving lane to the adjacent lane; When the degree of overlap between the vehicle region and the adjacent lane changes from being greater than the degree of overlap threshold to being less than the degree of overlap threshold, it is determined that the vehicle leaves the adjacent lane.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images