A method and device for detecting abnormal lane change of a vehicle

Abstract

The present application provides a detection method and device for abnormal lane change of vehicles. For a first road monitoring area, first data collected by a video image acquisition device and second data collected by a radar device are respectively obtained; the number of vehicles in the second data is greater than or equal to the number of vehicles in the first data; the second data includes the motion attribute information of the vehicle; based on the second data and the first data, the video attribute information of the vehicle in the first road monitoring area is determined; based on the video attribute information of the vehicle, the motion attribute information of the vehicle and the lane information in the first road monitoring area, the abnormal lane change of the vehicle in the first road monitoring area is determined. In this way, the second data collected by the radar device can correct the first data of the video image acquisition device to ensure the detection of abnormal lane change of vehicles at a longer distance.

Description

Background Art

[0002] Vehicle-road collaboration technology has promoted the innovative development of the automotive industry and can detect abnormal lane-changing behaviors of vehicles in real time, which is of great significance in improving traffic efficiency and safety. Usually, a pure video method is used to detect abnormal lane-changing behaviors of vehicles. However, due to the limited viewing range of video acquisition equipment and the fact that video acquisition equipment is easily affected by the natural environment (fog, heavy rain, etc.), the detection distance is short and it is not possible to determine whether the vehicle has made an abnormal lane change.

[0003] Based on this, a new method for detecting abnormal lane changes of vehicles is urgently needed to solve the above problems. Summary of the invention

[0004] The present application provides a method and device for detecting abnormal lane changes of vehicles, so as to improve the detection distance of abnormal lane changes of vehicles, and further improve the detection accuracy of abnormal lane changes.

[0005] In a first aspect, the present application provides a method for detecting abnormal lane change of a vehicle. The method can be performed by an electronic device, which can be a computing terminal, etc., and is performed as follows:

[0006] For the first road monitoring area, first data collected by the video image acquisition device and second data collected by the radar device are obtained respectively; the number of vehicles in the second data is greater than or equal to the number of vehicles in the first data; the second data includes motion attribute information of the vehicle; based on the second data and the first data, the video attribute information of the vehicle in the first road monitoring area is determined; based on the video attribute information of the vehicle, the motion attribute information of the vehicle and the lane information in the first road monitoring area, the abnormal lane change situation of the vehicle in the first road monitoring area is determined.

[0007] In the present application, for the first road monitoring area, the electronic device can collect first data based on the video image acquisition device and collect second data based on the radar device. Usually, the video image acquisition device is easily affected by the environment and its own line of sight, and the collection area may be short. The radar device locates the target (vehicle) by emitting electromagnetic waves, etc., and will not be affected by the environment and can collect targets at a longer distance. Based on this, the present application adjusts the first data collected by the video image acquisition device through the second data collected by the radar device. In this way, the first data can be corrected so that the electronic device can obtain more target information, thereby realizing the monitoring of abnormal lane changes of vehicles at a longer distance. Based on the vehicle's video attribute information, motion attribute information and lane information in the first road area, the electronic device can determine the abnormal lane change situation of each vehicle in the first road monitoring area. This method can improve the detection accuracy of abnormal lane changes.

[0008] In an optional manner, within the line of sight of the video image acquisition device, a first target vehicle is determined based on the video attribute information of the vehicle; based on the motion attribute information of the first target vehicle and the lane information in the first road monitoring area, it is determined whether the baseline of the first target vehicle intersects with the solid line of the lane; if so, it is determined that the first target vehicle has changed lanes abnormally.

[0009] The present application determines the abnormal lane change of the first target vehicle according to the intersection of the baseline of the first target vehicle and the solid line of the lane within the line of sight of the video image acquisition device. In this way, the detection accuracy and efficiency of abnormal lane change can be improved.

[0010] In an optional manner, within the line of sight of the video image acquisition device, a second target vehicle is determined based on the video attribute information of the vehicle; outside the line of sight of the video image acquisition device, the current motion attribute information of the second target vehicle is determined; based on the Kalman filter algorithm, a first position area where the second target vehicle is located in the next radar cycle is predicted; the second position area where the second target vehicle is located in the next radar cycle is obtained; if the first position area is different from the second position area, and the distance between the second position area and the solid line of the lane is less than the width threshold of the second target vehicle, then it is determined that the second target vehicle has changed lanes abnormally.

[0011] The present application determines the abnormal lane change of the second target vehicle outside the line of sight of the video image acquisition device based on the relationship between the predicted position area and the actual position area of ​​the second target vehicle in the next radar cycle. In this way, the detection accuracy and efficiency of abnormal lane change can be improved.

[0012] In an optional manner, based on the video attribute information of the vehicle, the abnormal lane-changing vehicle is determined to be a third target vehicle; the third target vehicle is one or more of the following: a police car, an ambulance, a fire truck, and an engineering vehicle; the latitude and longitude coordinates, vehicle type, and license plate information of the third target vehicle are determined.

[0013] This application takes into account that vehicles that change lanes abnormally may be special vehicles such as police cars and ambulances, and can be marked to remind smart vehicles that vehicles that change lanes abnormally are special vehicles and to be careful to avoid them when driving.

[0014] In an optional manner, the coordinate values ​​of the coordinate system of the radar device where the vehicle in the second data is located are converted into coordinate values ​​of the pixel coordinate system; the pixel coordinate system is the coordinate system where the first data is located; the second data is subjected to video fusion with the first data to determine the number of vehicles in the first road monitoring area and the area where the missing vehicles in the first data are located; data processing is performed on the area where the missing vehicles in the first data are located based on deep learning to determine the color information, vehicle model information, license plate information, etc. of the missing vehicles; based on the first data and the color information, vehicle model information, and license plate information of the missing vehicles, the video attribute information of the vehicles in the first road monitoring area is determined; the video attribute information includes: color information, vehicle model information, license plate information, quantity information, etc.

[0015] The present application converts the coordinate system of the radar device into the pixel coordinate system, and then fuses the second data with the first data to ensure that more vehicles can be detected in the first road monitoring area.

[0016] In an optional manner, the lane information in the first road monitoring area is determined based on a high-precision map of the first road monitoring area, or is determined by performing data analysis on image data of the first road monitoring area through a video image algorithm.

[0017] By acquiring lane information, it is possible to accurately determine whether the vehicle has abnormal lane-changing behavior, thereby improving data processing efficiency.

[0018] In an optional manner, the abnormal lane change of vehicles in the first road monitoring area is notified to the roadside equipment.

[0019] In this way, the roadside equipment can notify the smart vehicle of the vehicle's abnormal lane change, so as to remind the smart vehicle to avoid and ensure safe driving.

[0020] In an optional manner, the coordinate values ​​of the radar coordinate system where the vehicle is located in the second data are converted into coordinate values ​​of the world coordinate system through a first conversion matrix; the first conversion matrix is ​​determined based on the positional relationship between the radar device and the video image acquisition device; the coordinate values ​​of the world coordinate system are converted into coordinate values ​​of the coordinate system where the video image acquisition device is located through a second conversion matrix; the second conversion matrix is ​​different from the first conversion matrix; and the coordinate values ​​of the coordinate system where the video image acquisition device is located are converted into coordinate values ​​of the pixel coordinate system using a preset rule.

[0021] The coordinate values ​​of the second data in the pixel coordinate system determined in this way are more accurate and reliable.

[0022] In an optional manner, the coordinate values ​​of the coordinate system of the video image acquisition device are imaged and projected based on the focal length of the video image acquisition device to obtain the coordinate values ​​of the phase plane coordinate system of the video image acquisition device; based on the size of the pixels on the photosensitive chip of the video image acquisition device, the plane center of the phase plane coordinate system and the coordinate values ​​of the second data in the phase plane coordinate system, the coordinate values ​​of the second data in the pixel coordinate system after discretization are determined.

[0023] The coordinate values ​​of the second data in the pixel coordinate system determined in this way are more accurate and reliable.

[0024] In a second aspect, the present application provides a device for detecting abnormal lane change of a vehicle, comprising an acquisition unit, a first determination unit and a second determination unit.

[0025] Among them, the acquisition unit is used to acquire, for the first road monitoring area, the first data collected by the video image acquisition device and the second data collected by the radar device respectively; the number of vehicles in the second data is greater than or equal to the number of vehicles in the first data; the second data includes the motion attribute information of the vehicle; the first determination unit is used to determine the video attribute information of the vehicle in the first road monitoring area according to the second data and the first data; the second determination unit is used to determine the abnormal lane change situation of the vehicle in the first road monitoring area according to the video attribute information of the vehicle, the motion attribute information of the vehicle and the lane information in the first road monitoring area.

[0026] In the present application, for the first road monitoring area, the electronic device can collect first data based on the video image acquisition device and collect second data based on the radar device. Usually, the video image acquisition device is easily affected by the environment and its own line of sight, and the collection area may be short. The radar device locates the target (vehicle) by emitting electromagnetic waves, etc., and will not be affected by the environment and can collect targets at a longer distance. Based on this, the present application adjusts the first data collected by the video image acquisition device through the second data collected by the radar device. In this way, the first data can be corrected so that the electronic device can obtain more target information, thereby realizing the monitoring of abnormal lane changes of vehicles at a longer distance. Based on the vehicle's video attribute information, motion attribute information and lane information in the first road area, the electronic device can determine the abnormal lane change situation of each vehicle in the first road monitoring area. This method can improve the detection accuracy of abnormal lane changes.

[0027] In an optional manner, the second determination unit is specifically used to determine the first target vehicle within the line of sight of the video image acquisition device based on the video attribute information of the vehicle; determine whether the baseline of the first target vehicle intersects with the solid line of the lane based on the motion attribute information of the first target vehicle and the lane information in the first road monitoring area; if so, determine that the first target vehicle has changed lanes abnormally.

[0028] In an optional manner, the second determination unit is specifically used to determine the second target vehicle within the line of sight of the video image acquisition device based on the video attribute information of the vehicle; determine the current motion attribute information of the second target vehicle outside the line of sight of the video image acquisition device; predict the first position area where the second target vehicle is located in the next radar cycle based on the Kalman filter algorithm; obtain the second position area where the second target vehicle is located in the next radar cycle; if the first position area is different from the second position area, and the distance between the second position area and the solid line of the lane is less than the width threshold of the second target vehicle, then determine that the second target vehicle has changed lanes abnormally.

[0029] In an optional manner, the second determination unit is also used to determine that the abnormal lane-changing vehicle is a third target vehicle based on the video attribute information of the vehicle; the third target vehicle is one or more of the following: a police car, an ambulance, a fire truck, and an engineering vehicle; and determine the latitude and longitude coordinates, vehicle type, and license plate information of the third target vehicle.

[0030] In an optional manner, the first determination unit is specifically used to convert the coordinate values ​​of the coordinate system of the radar device where the vehicle in the second data is located into the coordinate values ​​of the pixel coordinate system; the pixel coordinate system is the coordinate system where the first data is located; the second data is subjected to video fusion with the first data to determine the number of vehicles in the first road monitoring area and the area where the missing vehicles in the first data are located; based on deep learning, data processing is performed on the area where the missing vehicles in the first data are located to determine the color information, vehicle model information, and license plate number information of the missing vehicles; based on the first data and the color information, vehicle model information, and license plate number information of the missing vehicles, the video attribute information of the vehicles in the first road monitoring area is determined; the video attribute information includes: color information, vehicle model information, quantity information, and license plate number information, etc.

[0031] In an optional manner, the lane information in the first road monitoring area is determined based on a high-precision map of the first road monitoring area, or is determined by performing data analysis on image data of the first road monitoring area through a video image algorithm.

[0032] In an optional manner, the device further includes a notification unit, configured to notify the roadside equipment of abnormal lane changes of vehicles in the first road monitoring area.

[0033] In an optional manner, the first determination unit is specifically used to convert the coordinate values ​​of the radar coordinate system where the vehicle is located in the second data into coordinate values ​​of the world coordinate system through a first conversion matrix; the first conversion matrix is ​​determined based on the positional relationship between the radar device and the video image acquisition device; the coordinate values ​​of the world coordinate system are converted into coordinate values ​​of the coordinate system where the video image acquisition device is located through a second conversion matrix; the second conversion matrix is ​​different from the first conversion matrix; and a preset rule is used to convert the coordinate values ​​of the coordinate system where the video image acquisition device is located into coordinate values ​​of the pixel coordinate system.

[0034] In an optional manner, the first determination unit is specifically used to perform imaging projection on the coordinate values ​​of the coordinate system where the video image acquisition device is located based on the focal length of the video image acquisition device to obtain the coordinate values ​​of the phase plane coordinate system of the video image acquisition device; based on the size of the pixels on the photosensitive chip of the video image acquisition device, the plane center of the phase plane coordinate system and the coordinate values ​​of the second data in the phase plane coordinate system, determine the coordinate values ​​of the second data in the pixel coordinate system after discretization.

[0035] In a third aspect, the present application provides a computing device, comprising: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the method described in the first aspect according to the obtained program.

[0036] In a fourth aspect, the present application provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the method described in the first aspect.

[0037] For the technical effects that can be achieved in the above-mentioned second to fourth aspects, please refer to the description of the technical effects that can be achieved by the corresponding possible design schemes in the above-mentioned first aspect, and this application will not repeat them here.

[0038] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a device layout;

[0041] Figure 2 A flow chart of a method for detecting abnormal lane change of a vehicle provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a coordinate conversion process provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of coordinate transformation provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the conversion between the camera coordinate system and the phase plane coordinate system provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of vehicle detection provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of vehicle detection provided in an embodiment of the present application;

[0047] Figure 8-1 A schematic diagram of vehicle detection provided in an embodiment of the present application;

[0048] Figure 8-2 A schematic diagram of vehicle detection provided in an embodiment of the present application;

[0049] Fig. 9 A schematic flow chart of another method for detecting abnormal lane change of a vehicle provided in an embodiment of the present application;

[0050] Fig.10 A schematic diagram of the structure of a device for detecting abnormal lane change of a vehicle provided in an embodiment of the present application;

[0051] Fig.11 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0053] It should be noted that the terms "first", "second", etc. in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] As described in the background technology, existing solutions mainly use video algorithms to implement abnormal vehicle lane change detection, which faces a series of practical difficulties and defects.

[0055] 1) Environmental factors such as lighting and weather have a significant impact on the detection accuracy of video algorithms. The image features of the same vehicle target will change dramatically under different lighting and weather conditions. Therefore, the vehicle target video detection algorithm often misses or misdetects the target, resulting in low accuracy in subsequent abnormal lane change detection.

[0056] 2) The effective detection distance of the pure video solution is affected by the algorithm design. The optimal detection distance of the video is only 50 to 70 meters. Vehicle targets with smaller pixels in the image are not easily detected by the video algorithm, which greatly limits the application distance of the video algorithm.

[0057] 3) The speed and distance measurement capabilities of the video algorithm are poor, and it is unable to accurately obtain the movement information of vehicles that change lanes abnormally. In the vehicle-road collaborative scenario, it is unable to provide accurate traffic event location coordinates for other road traffic participants.

[0058] In order to solve the above-mentioned problem, the present application proposes a new method for detecting abnormal lane change of a vehicle.

[0059] The following is a detailed description of the detection process of abnormal lane change of vehicles. In the following embodiments of the present application, "and / or" describes the association relationship of the associated objects, indicating that there may be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or its similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. The singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear indication to the contrary in the context. Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. For example, the first task execution device and the second task execution device are only used to distinguish different task execution devices, and do not indicate the difference in priority or importance of the two task execution devices.

[0060] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] like Figure 1 As shown in the figure, millimeter-wave radars and video cameras are deployed on urban roads, and edge computing terminals (multi-acess edge computing, MEC) obtain video images collected by cameras and data collected by millimeter-wave radars in real time. Millimeter-wave radars are radars that work in the millimeter-wave frequency band. Millimeter-wave radars can actively transmit electromagnetic wave signals and receive echoes. Based on the time difference between transmitting and receiving electromagnetic wave signals, the relative distance, relative speed, and relative direction of the vehicle target are obtained. Usually, millimeter-wave radars and video cameras can be deployed on the same horizontal plane, such as Figure 1As shown, it may also be other positions, which are not specifically limited in this application. MEC can perform data analysis based on the video images collected by the camera and the data collected by the millimeter wave radar to determine the abnormal lane change of the vehicle.

[0062] like Figure 2 As shown, the present application provides a method for detecting abnormal lane change of a vehicle, which can be executed by an electronic device or by a server, and the present application does not specifically limit it, such as the above-mentioned MEC, etc. Here, taking an electronic device as an example, the following can be executed:

[0063] Step 201, for the first road monitoring area, respectively obtain the first data collected by the video image acquisition device and the second data collected by the radar device; the number of vehicles in the second data is greater than or equal to the number of vehicles in the first data; the second data includes the motion attribute information of the vehicle.

[0064] It should be noted that the field of view of the video image acquisition device is limited, such as a visual range of 50 to 70 meters, so the video acquisition device can only collect video image information within the visual range. In the case of poor environmental quality (fog, heavy rain and other weather conditions), it may only be able to collect video image information within a range of 10 to 20 meters. However, the radar device determines whether there is a vehicle by emitting electromagnetic wave signals, so it will not be affected by the environment. The range of collected data may be 100 to 200 meters, etc. Compared with the video image acquisition device, it can collect vehicle information at a farther range, but the radar device cannot obtain information such as the color, model, and license plate number of the vehicle. Therefore, in the present application, in the first road monitoring area, when the video image acquisition device and the radar device are deployed at the same time, the number of vehicles in the first data collected by the video image acquisition device is less than or equal to the number of vehicles in the second device collected by the radar device.

[0065] Step 202: Determine video attribute information of vehicles in the first road monitoring area based on the second data and the first data.

[0066] It should be noted that the number of vehicles included in the second data may be greater than the number of vehicles included in the first data. By fusing the vehicles in the second data with the vehicles in the first data, more vehicle information can be obtained, and the video attribute information such as the vehicle's color, model, license plate, etc. can be determined based on the image information collected by the video image acquisition device.

[0067] Optionally, the electronic device may convert the coordinate values ​​of the coordinate system of the radar device where the vehicle is located in the second data into coordinate values ​​of the pixel coordinate system; the pixel coordinate system is the coordinate system where the first data is located. Specifically, the second data detected by the radar device is mapped to the pixel coordinate system through coordinate conversion to achieve spatial synchronization of the radar target and the video target. The coordinate conversion process is as follows: Figure 3As shown. The coordinate system of the radar device is transformed into the world coordinate system through rotation and translation, the world coordinate system is transformed into the camera coordinate system through rotation and translation, the camera coordinate system is transformed into the phase plane coordinate system through projection imaging, and the phase plane coordinate system is transformed into the pixel coordinate system through discretization.

[0068] Optionally, the electronic device can convert the coordinate values ​​of the radar coordinate system where the vehicle is located in the second data into coordinate values ​​of the world coordinate system through a first conversion matrix; the first conversion matrix is ​​determined based on the positional relationship between the radar device and the video image acquisition device (that is, the following formula 1, since the video camera and the radar device are on the same horizontal plane, there is no angular rotation, so the conversion matrix only includes a translation matrix, not a rotation matrix); the coordinate values ​​of the world coordinate system are converted into coordinate values ​​of the coordinate system where the video image acquisition device is located through a second conversion matrix (that is, the following formula 2, which can be understood by referring to the description below and will not be elaborated here); the second conversion matrix is ​​different from the first conversion matrix; and the coordinate values ​​of the coordinate system where the video image acquisition device is located are converted into coordinate values ​​of the pixel coordinate system using preset rules (wherein the preset rules may include conversion rules between phase plane and pixel plane coordinates and preset logical rules, etc., which are not specifically limited here).

[0069] There are many ways to deploy radar equipment and video image acquisition equipment. Figure 1 Taking the layout method in as an example, the corresponding position relationship between the radar device coordinate system and the video camera coordinate system is as follows Figure 4 Assume O l is the origin of the radar equipment's coordinate system, O w The origin of the world coordinate system is centered on the video camera (i.e., the video image acquisition device). First, the coordinates in the radar device's coordinate system are converted to the world coordinate system, where the radar device can obtain the x-axis and y-axis coordinate information of the target (i.e., the vehicle), but cannot obtain the z-axis coordinate information of the target. Therefore, the coordinate system of the radar device can be converted to the world coordinate system. l To the world coordinate system O w The transformation of is considered as the transformation of the two-dimensional xy coordinate system. l and O w The relationship between is realized through the transformation matrix (also known as the first transformation matrix). The transformation matrix consists of two parts, one is the rotation matrix caused by the angle, and the other is the translation matrix generated by the translation. Among them, the translation amount of the translation matrix can be understood as the coordinates of the radar device in the world coordinate system, which is the following Figure 4The negative number -D of the distance D from the radar device to the video camera. In this application, the radar device and the video camera are installed in the same horizontal plane, so it can be understood that the rotation angle α of the radar coordinate system relative to the world coordinate system in the rotation matrix is ​​0, as shown in the following formula 1. Where R2 represents the rotation matrix from the above radar coordinate system to the above world coordinate system, and t2 represents the translation matrix from the above radar coordinate system to the above world coordinate system; (x l ,y l ) represents the coordinate position of the second data in the radar coordinate system, (x w ,y w ) represents the coordinate position of the second data in the above-mentioned world coordinate system after the coordinate transformation is performed by the above-mentioned rotation matrix R2 and the above-mentioned translation matrix t2.

[0070]

[0071] There are four plane coordinate systems in the camera model: pixel plane coordinate system (u, v), phase plane coordinate system (image physical coordinate system (x, y)), camera coordinate system (that is, the coordinate system where the video image acquisition device is located) (x c ,y c ,z c ) and the world coordinate system (x w ,y w ,z w ). Since the radar device cannot obtain the z-axis coordinate information of the target, after the above-mentioned coordinate conversion from the radar coordinate system to the world coordinate system, the coordinate position in the world coordinate system obtained does not have the z-axis coordinate information. w The value is assumed using the prior knowledge of the vehicle height. Assuming that the height of the vehicle target is H = 1.8m, then z w =1.8. Thus, the coordinates of the second data in the world coordinate system (x w ,y w ,z w ).

[0072] The relationship between the world coordinate system and the camera coordinate system can also be obtained through the transformation matrix (also known as the second transformation matrix), that is, the transformation from the world coordinate system to the camera coordinate system is achieved through the rotation matrix R1 and the translation matrix t1. As shown in the following formula 2. Where R1 represents the rotation matrix from the world coordinate system to the camera coordinate system, and t1 represents the translation matrix from the world coordinate system to the camera coordinate system; (x w ,y w ,z w ) represents the coordinate position of the second data in the above world coordinate system, (x c ,y c ,z c) represents the coordinate position of the second data in the camera coordinate system after the rotation matrix R1 and the translation matrix t1 are used for coordinate transformation. The angle of rotation of the world coordinate system around the x-axis is α1, the angle of rotation around the y-axis is β1, the angle of rotation around the z-axis is θ1, and the translation distance on the x-axis is t x , the translation distance on the y-axis is t y , the translation distance on the z-axis is t z .

[0073]

[0074]

[0075]

[0076] Optionally, the electronic device may perform imaging projection on the coordinate values ​​of the coordinate system where the video image acquisition device is located based on the focal length of the video image acquisition device to obtain the coordinate values ​​of the phase plane coordinate system of the video image acquisition device (that is, the coordinate position of the phase plane is obtained by projecting the focal length of the video camera and the coordinate position of the second data in the camera coordinate system in the manner provided by Formula 3 as described below); based on the size of pixels on the photosensitive chip of the video image acquisition device, the plane center of the phase plane coordinate system and the coordinate value of the second data in the phase plane coordinate system, determine the coordinate value of the second data after discretization in the pixel coordinate system (that is, the coordinate position of the phase plane is obtained by discretizing the actual size of pixels on the photosensitive chip of the video camera, the plane center of the phase plane coordinate system and the coordinate value of the second data in the phase plane coordinate system in the manner provided by Formula 4 as described below).

[0077] as follows Figure 5 As shown in the figure, there is an imaging projection relationship between the camera coordinate system and the phase plane coordinate system, which is closely related to the focal length f of the camera. As shown in the following formula 3. f represents the focal length of the video camera; (x c ,y c ,z c ) represents the coordinate position of the second data in the above-mentioned camera coordinate system, and (x, y) represents the coordinate position of the second data in the phase plane coordinate system after imaging projection.

[0078]

[0079] The phase plane coordinate system is converted to the pixel coordinate system through discretization. As shown in the following formula 4. dx and dy represent the actual size of the pixel on the photosensitive chip of the video camera; u0 and v0 represent the plane center of the phase plane coordinate system, (x, y) represents the coordinate position of the second data in the above phase plane coordinate system, and (u, v) represents the coordinate position of the second data in the pixel coordinate system after discretization.

[0080]

[0081] In the present application, the above-mentioned coordinate conversion parameters can be obtained by calibrating the radar device and the video camera, and the radar coordinates of the second data collected by the radar device can be converted into a pixel coordinate system through the above-mentioned parameters.

[0082] Afterwards, the electronic device can perform video fusion of the second data with the first data to determine the number of vehicles in the first road monitoring area and the area where the missing vehicles in the first data are located; perform data processing on the area where the missing vehicles in the first data are located based on deep learning to determine the color information, vehicle model information, license plate number information, etc. of the missing vehicles; determine the video attribute information of the vehicles in the first road monitoring area based on the first data and the color information, vehicle model information, and license plate information of the missing vehicles; the video attribute information includes: color information, vehicle model information, quantity information, and license plate number information, etc.

[0083] by Figure 6 For example, Figure 6 In 6-1, there are 6 target vehicles Car1, Car2...Car6 within the effective range of the radar equipment; Figure 6 The 6-2 millimeter-wave radar detects six radar target vehicles Car1, Car2, ... Car6 on the road by actively transmitting electric waves; due to possible reasons such as lighting, weather, background pixels, camera resolution, etc. Figure 6 In 6-3, within the effective range of the video camera, the video algorithm only detects two video target vehicles, Car5 and Car6; through the above coordinate transformation, Figure 6 In 6-4, the radar target vehicle and the video target vehicle are mapped to the same pixel coordinate system. The video processing unit can perform multi-target detection and positioning on the video image based on the deep learning target detection algorithm. The deep learning target detection algorithm mentioned in this application can be a single-stage target detection algorithm such as the you only look once (YOLO) series (i.e., one of the target detection algorithms), single-stage multi-frame detection (single shot multibox detector, SSD), or a two-stage target detection algorithm such as faster region convolutional neural networks (Faster R-CNN). This application can use the YOLOV5 target detection algorithm to complete vehicle target detection in the video image. The YOLOV5 target detection algorithm can return the type of vehicle detected in the video image and the position coordinates of the vehicle target in the video image and the pixel length and width of the vehicle target.

[0084] The radar device includes a radio frequency module and an array antenna module. The radio frequency module transmits an electromagnetic beam, and the array antenna module receives the returned electromagnetic beam. The point cloud data of the millimeter wave radar can be obtained through the returned electromagnetic beam. The point cloud data of the same target vehicle is clustered, and the midpoint of all the point cloud data of the target vehicle is taken as the center of mass of the target vehicle, so as to determine the x coordinate (x pos ) and the y coordinate (y pos ), take the minimum and maximum values ​​of the y coordinates in the point cloud data to get the vehicle length of the target vehicle. Figure 6 As shown in Figure 6-4, the target vehicle Car4 in the dotted box (i.e., the area where the missing vehicle is located) is not detected by the video algorithm. In order to identify the type of the target vehicle, the present application calculates the x coordinate (x pos ) and the y coordinate (y pos ) is converted to the pixel coordinate system to obtain the x′ coordinate (x′) of the target vehicle in the pixel coordinate system. pos ) and y′ coordinate (y′ pos ), with (x′ pos ,y′ pos ) as the center point, and the pixel length length′ corresponding to the vehicle length length in the pixel coordinate system as the side length, intercept the image area in the pixel coordinate system. The type of the target vehicle in the area is identified based on the deep learning vehicle type detection network. This ensures that the algorithm can obtain the types of all target vehicles within the effective range of the video on the road.

[0085] In addition, in order to obtain more abundant video attribute information of vehicles, the present application simultaneously intercepts the image area where Car4, Car5, and Car6 are located, and identifies the attribute information such as vehicle color, vehicle logo, and license plate number through the target vehicle attribute detection network. The vehicle type detection network and vehicle attribute detection network described in the present application can be lightweight neural networks such as MobileNet and ShuffleNet, or large and medium-sized networks such as ResNet. In order to achieve better detection effect, ResNet50 can be used as the backbone network of the vehicle type detection network and the vehicle attribute detection network in the present application to obtain vehicle visual attribute information such as vehicle type, vehicle color, and vehicle logo. For example, the ResNet50 network model can be trained in advance by a vehicle picture sample set containing different colors (for example, black, white, blue, and brown). The image area of ​​the missing vehicle Car4 is input into the trained ResNet50 network model, and the ResNet50 network model extracts features from the image area of ​​the missing vehicle Car4, and outputs the probability that the image area of ​​the missing vehicle Car4 belongs to black, white, blue, and brown respectively. When the probability that Car4 is black is the highest, it is determined that Car4 is black.

[0086] In addition, under good environmental conditions, optical character recognition (OCR) technology may also be used to identify the license plate number of the target vehicle, which is not specifically limited in this application.

[0087] Step 203: determining an abnormal lane change situation of a vehicle in the first road monitoring area according to the video attribute information of the vehicle, the motion attribute information of the vehicle and the lane information in the first road monitoring area.

[0088] Optionally, the lane information in the first road monitoring area is determined based on a high-precision map of the first road monitoring area, or is determined by performing data analysis on image data of the first road monitoring area through a video image algorithm.

[0089] In actual application, a high-precision map can be pre-recorded in the electronic device. The high-precision map records in detail the lane line type on the road and the high-precision coordinates of the starting point of the lane solid line area. If the high-precision map is not pre-recorded in the electronic device, the lane line in the video image can be detected in real time based on the deep learning algorithm to determine the lane information. Among them, the lane line type can be a white solid line, a single yellow solid line, a double yellow solid line, a yellow dashed line, a white dashed line, and other lane line types that prohibit abnormal lane changes. At the same time, the coordinate position of the lane solid line area in the pixel point coordinate system can also be obtained based on the video algorithm.

[0090] The present application can detect the pixel width and pixel length of the target vehicle in the pixel coordinate system by using the target detection algorithm within the line of sight of the video image acquisition device. Assuming that the pixel length of the detected target vehicle is 90 pixels and the pixel width is 36 pixels, and the radar device detects that the length of the target vehicle is 4.3 meters, then the actual width of the target vehicle is approximately 4.3*36 / 90=1.72m.

[0091] If the target detection algorithm does not detect the pixel width and pixel length of the target vehicle in the pixel coordinate system within the viewing range of the video image acquisition device, such as Figure 6-4 In the Car4, since the vehicle type of Car4 has been detected in the above vehicle type detection network, assuming that Car4 is detected as a sedan, the length of a sedan is generally between 3.8 meters and 4.3 meters, and the width is generally between 1.6 meters and 1.8 meters. If the radar device detects that the length of the target vehicle Car4 is 4.0 meters, based on the above sedan size range, it can be inferred that the actual width of the target vehicle Car4 is about 1.68 meters; since the pixel length length′ of the target vehicle Car4 in the pixel coordinate system has been obtained in the above coordinate conversion process, assuming that length′=88, the pixel width of the target vehicle Car4 in the pixel coordinate system is about 88*1.68 / 4≈37, and it can be estimated that the pixel length of Car4 in the pixel coordinate system is about 37. Therefore, the actual vehicle width of the target vehicle and the pixel width in the pixel coordinate system can be obtained according to the first data and the second data.

[0092] Optionally, the electronic device determines the first target vehicle within the line of sight of the video image acquisition device based on the video attribute information of the vehicle; determines whether the baseline of the first target vehicle intersects with the solid line of the lane based on the motion attribute information of the first target vehicle and the lane information in the first road monitoring area; if so, it determines that the first target vehicle has changed lanes abnormally.

[0093] as follows Figure 7As shown in the figure, the coordinate points in the figure are the coordinate points of the lane solid line area in the pixel coordinate system detected by the high-precision map or video algorithm. In the video image of video frame frame1, the radar-vision fusion algorithm detects that the lane where the target vehicle CarA is located is the second lane on the left, and obtains the coordinate points of the target vehicle in the pixel coordinate system. In the video image of video frame frame2, it is detected that the baseline of the target vehicle CarA is pressed into the lane dotted line, and no abnormal lane change occurs. In the video image of video frame frame3, it is detected that the lane where the target vehicle CarA is located is the third lane on the left. When the target vehicle CarA changes lanes, it is calculated whether the line connecting the pixel width positioning coordinates on both sides of the target vehicle body intersects with the line connecting the positioning coordinates of the lane solid line area. When the two do not intersect, the abnormal lane change alarm of the target vehicle is not triggered. In the video image of video frame frame4, it is detected that the baseline of the target vehicle CarA is pressed into the lane solid line, and the line connecting the pixel width positioning coordinates on both sides of the target vehicle body intersects with the line connecting the positioning coordinates of the lane solid line area, and the abnormal lane change alarm of the target vehicle is triggered.

[0094] In the detection of abnormal lane change of vehicles, the video attribute information of the target vehicle can be used to determine whether the abnormal lane change vehicles in frame1, frame2, frame3 and frame4 video frames are vehicles with the same ID. When the license plate number of the target vehicle detected in the above video frame is the same license plate number, it is determined that a vehicle with a certain ID has undergone abnormal lane change behavior. When the license plate number of the target vehicle cannot be detected in the above video frame, the vehicle re-identification algorithm is used to determine that the lane-changing vehicle is the same vehicle based on the appearance features of the target vehicle image area. When it is determined that the vehicle has abnormal lane change behavior, the electronic device records the high-precision positioning of the target vehicle when the abnormal lane change occurs, that is, the longitude and latitude coordinates, and uploads the attribute information such as vehicle type, body color, vehicle model, license plate number, and vehicle speed to the vehicle-road collaborative cloud platform, and simultaneously sends it to the road side unit (RSU). The RSU broadcasts the traffic event to the intelligent vehicles in the Internet of Vehicles.

[0095] The present application determines the abnormal lane change of the first target vehicle according to the intersection of the baseline of the first target vehicle and the solid line of the lane within the line of sight of the video image acquisition device. In this way, the detection accuracy and efficiency of abnormal lane change can be improved.

[0096] Optionally, the electronic device determines the second target vehicle based on the video attribute information of the vehicle within the line of sight of the video image acquisition device; determines the motion attribute information of the current second target vehicle outside the line of sight of the video image acquisition device (and within the detection range of the radar device); predicts the first position area where the second target vehicle will be located in the next radar cycle based on the Kalman filter algorithm; obtains the second position area where the second target vehicle will be located in the next radar cycle; if the first position area is different from the second position area, and the distance between the second position area and the solid line of the lane is less than the width threshold of the second target vehicle, then it is determined that the second target vehicle has changed lanes abnormally.

[0097] as follows Figure 8-1 As shown, the electronic device predicts the possible location area of ​​the target vehicle in the radar coordinate system in the next radar cycle based on the Kalman filter algorithm according to the motion properties such as the speed of the target vehicle in the current radar cycle, such as Figure 8-1 As shown, the solid-line frame is the actual position area where the target vehicles CarA and CarB are located in the current radar cycle, and the dotted-line frame is the position area where the target vehicles CarA and CarB may reach in the next radar cycle predicted by the electronic device based on the Kalman filter algorithm. In the next radar cycle, when the radar device detects that the target vehicles CarA and CarB appear at the position predicted in the previous radar cycle, it is determined that the target vehicles have not made abnormal lane change behavior.

[0098] as follows Figure 8-2 As shown in the figure, the solid-line frame is the actual position area where the target vehicles CarA, CarB, and CarC are located in the current radar cycle, and the dotted-line frame is the position area in the radar coordinate system where the target vehicles CarA, CarB, and CarC may reach in the next radar cycle predicted by the electronic device based on the Kalman filter algorithm. In the next radar cycle, the radar device detects that the target vehicles CarB and CarC appear at the positions predicted in the previous radar cycle, while the target vehicle CarA does not appear at the predicted position (such as Figure 8-2 ), a target vehicle appears in the middle of the adjacent left first lane and second lane, and the distance between the target vehicle and the left second lane is less than 1 / 2 of the vehicle width, then it is determined that the target vehicle CarA has an abnormal lane change behavior.

[0099] Similarly, when it is determined that a vehicle has made an abnormal lane change, the electronic device records the high-precision positioning of the target vehicle when the abnormal lane change occurs, that is, the longitude and latitude coordinates, and uploads the vehicle type, body color, vehicle model, license plate number, vehicle speed and other attribute information detected within the effective range of the video to the vehicle-road collaborative cloud platform, and simultaneously sends it to the roadside unit (RSU). The RSU broadcasts the traffic event to the smart vehicles in the Internet of Vehicles.

[0100] The present application determines the abnormal lane change of the second target vehicle outside the line of sight of the video image acquisition device based on the relationship between the predicted position area and the actual position area of ​​the second target vehicle in the next radar cycle. In this way, the detection accuracy and efficiency of abnormal lane change can be improved.

[0101] In addition, the electronic device can also determine that the abnormal lane-changing vehicle is a third target vehicle (i.e., a special vehicle) based on the vehicle's video attribute information; the third target vehicle is one or more of the following: police cars, ambulances, fire trucks, and engineering vehicles; determine the latitude and longitude coordinates, vehicle type, and license plate information of the third target vehicle.

[0102] When a vehicle target is detected within the effective range of the video with its hazard lights on and an abnormal lane change occurs, the high-precision positioning of the lane change, i.e., the longitude and latitude coordinates, as well as information such as vehicle type, body color, vehicle model, license plate number, vehicle speed, and the status of the hazard lights on the vehicle are recorded. As special road traffic participants, special vehicles and faulty vehicles will be urgently broadcast to smart vehicles through C-V2X technology when abnormal lane changes occur, reminding the vehicles to make emergency avoidance, thus ensuring the driving safety of smart vehicles.

[0103] The present application notifies the roadside equipment of the abnormal lane change of the vehicle in the first road monitoring area, so that the roadside equipment can notify the intelligent vehicle of the abnormal lane change of the vehicle to remind the intelligent vehicle to avoid and ensure safe driving.

[0104] This application uses radar and video fusion to monitor abnormal lane-changing behavior of vehicles in real time. The technical solution is as follows Fig. 9 shown.

[0105] The edge computing terminal accesses the video signal, and identifies and locates the vehicle targets in the video image based on the deep learning algorithm, identifies and locates the vehicle targets in the urban road based on the millimeter wave radar, and determines the lane information based on the video image. After that, the coordinates of the radar target and the video target are fused, which can be referred to the description above and will not be repeated here. The video target is corrected based on the radar target to obtain the attribute information of the video target. Based on the radar target as the benchmark, the image target area is extracted for the missed video target, and then the visual attribute of all the corrected vehicle targets is extracted based on the deep learning algorithm. After that, the lane line type in the video image is detected based on the deep learning algorithm, and the high-precision positioning module is used to complete the high-precision positioning of the lane solid line point and the vehicle target. Within the effective range of the video, based on the high-precision positioning information, the visual attributes of the vehicle target and the motion attributes of the vehicle target, the position information of the same target vehicle is monitored in real time. When the same target vehicle crosses the solid line of the lane, the visual attribute information and high-precision positioning information of the target vehicle are recorded to complete the detection of abnormal lane change behavior of the vehicle. Outside the effective range of the video, the radar target is tracked and predicted, and the position information of the same target vehicle is monitored in real time to detect abnormal lane change behavior of vehicles outside the effective range of the video. The deep learning algorithm is used to detect the type of special vehicles, the status of the warning lights of special vehicles, and the status of the double flash lights of faulty vehicles in real time. Through real-time monitoring of special vehicles such as police cars, ambulances, fire trucks, engineering vehicles, and faulty vehicles, comprehensive monitoring of abnormal lane change behavior of vehicles is achieved, ensuring the driving safety of road vehicles.

[0106] In the present application, for the first road monitoring area, the electronic device can collect first data based on the video image acquisition device and collect second data based on the radar device. Usually, the video image acquisition device is easily affected by the environment and its own line of sight, and the collection area may be short. The radar device locates the target (vehicle) by emitting electromagnetic waves, etc., and will not be affected by the environment and can collect targets at a longer distance. Based on this, the present application adjusts the first data collected by the video image acquisition device through the second data collected by the radar device. In this way, the first data can be corrected so that the electronic device can obtain more target information, thereby realizing the monitoring of abnormal lane changes of vehicles at a longer distance. Based on the vehicle's video attribute information, motion attribute information and lane information in the first road area, the electronic device can determine the abnormal lane change situation of each vehicle in the first road monitoring area. This method can improve the detection accuracy of abnormal lane changes.

[0107] Based on the same concept, the embodiment of the present application provides a detection device for abnormal lane change of a vehicle, such as Fig.10 As shown, it includes: an acquisition unit 1001, a first determination unit 1002 and a second determination unit 1003.

[0108] Among them, the acquisition unit 1001 is used to acquire, for the first road monitoring area, the first data collected by the video image acquisition device and the second data collected by the radar device; the number of vehicles in the second data is greater than or equal to the number of vehicles in the first data; the second data includes the motion attribute information of the vehicle; the first determination unit 1002 is used to determine the video attribute information of the vehicle in the first road monitoring area according to the second data and the first data; the second determination unit 1003 is used to determine the abnormal lane change situation of the vehicle in the first road monitoring area according to the video attribute information of the vehicle, the motion attribute information of the vehicle and the lane information in the first road monitoring area.

[0109] In the present application, for the first road monitoring area, the electronic device can collect first data based on the video image acquisition device and collect second data based on the radar device. Usually, the video image acquisition device is easily affected by the environment and its own line of sight, and the collection area may be short. The radar device locates the target (vehicle) by emitting electromagnetic waves, etc., and will not be affected by the environment and can collect targets at a longer distance. Based on this, the present application adjusts the first data collected by the video image acquisition device through the second data collected by the radar device. In this way, the first data can be corrected so that the electronic device can obtain more target information, thereby realizing the monitoring of abnormal lane changes of vehicles at a longer distance. Based on the vehicle's video attribute information, motion attribute information and lane information in the first road area, the electronic device can determine the abnormal lane change situation of each vehicle in the first road monitoring area. This method can improve the detection accuracy of abnormal lane changes.

[0110] In an optional manner, the second determination unit 1003 is specifically used to determine the first target vehicle within the line of sight of the video image acquisition device based on the video attribute information of the vehicle; determine whether the baseline of the first target vehicle intersects with the solid line of the lane based on the motion attribute information of the first target vehicle and the lane information in the first road monitoring area; if so, determine that the first target vehicle has changed lanes abnormally.

[0111] The present application determines the abnormal lane change of the first target vehicle according to the intersection of the baseline of the first target vehicle and the solid line of the lane within the line of sight of the video image acquisition device. In this way, the detection accuracy and efficiency of abnormal lane change can be improved.

[0112] In an optional manner, the second determination unit 1003 is specifically used to determine the second target vehicle within the line of sight of the video image acquisition device based on the video attribute information of the vehicle; determine the current motion attribute information of the second target vehicle outside the line of sight of the video image acquisition device; predict the first position area where the second target vehicle is located in the next radar cycle based on the Kalman filter algorithm; obtain the second position area where the second target vehicle is located in the next radar cycle; if the first position area is different from the second position area, and the distance between the second position area and the solid line of the lane is less than the width threshold of the second target vehicle, then determine that the second target vehicle has changed lanes abnormally.

[0113] The present application determines the abnormal lane change of the second target vehicle outside the line of sight of the video image acquisition device based on the relationship between the predicted position area and the actual position area of ​​the second target vehicle in the next radar cycle. In this way, the detection accuracy and efficiency of abnormal lane change can be improved.

[0114] In an optional manner, the second determination unit 1003 is also used to determine that the abnormal lane-changing vehicle is a third target vehicle based on the video attribute information of the vehicle; the third target vehicle is one or more of the following: a police car, an ambulance, a fire truck, and an engineering vehicle; and determine the latitude and longitude coordinates, vehicle type, and license plate information of the third target vehicle.

[0115] This application takes into account that vehicles that change lanes abnormally may be special vehicles such as police cars and ambulances, and can be marked to remind smart vehicles that vehicles that change lanes abnormally are special vehicles and to be careful to avoid them when driving.

[0116] In an optional manner, the first determination unit 1002 is specifically used to convert the coordinate value of the coordinate system of the radar device where the vehicle in the second data is located into the coordinate value of the pixel coordinate system; the pixel coordinate system is the coordinate system where the first data is located; the second data is subjected to video fusion with the first data to determine the number of vehicles in the first road monitoring area and the area where the missing vehicles in the first data are located; based on deep learning, data processing is performed on the area where the missing vehicles in the first data are located to determine the color information, vehicle model information, and license plate number information of the missing vehicles; based on the first data and the color information, vehicle model information, and license plate number information of the missing vehicles, the video attribute information of the vehicles in the first road monitoring area is determined; the video attribute information includes: color information, vehicle model information, quantity information, and license plate number information, etc.

[0117] The present application converts the coordinate system of the radar device into the pixel coordinate system, and then fuses the second data with the first data to ensure that more vehicles can be detected in the first road monitoring area.

[0118] In an optional manner, the lane information in the first road monitoring area is determined based on a high-precision map of the first road monitoring area, or is determined by performing data analysis on image data of the first road monitoring area through a video image algorithm.

[0119] By acquiring lane information, it is possible to accurately determine whether the vehicle has abnormal lane-changing behavior, thereby improving data processing efficiency.

[0120] In an optional manner, the device further includes a notification unit, configured to notify the roadside equipment of abnormal lane changes of vehicles in the first road monitoring area.

[0121] In this way, the roadside equipment can notify the smart vehicle of the vehicle's abnormal lane change to remind the smart vehicle to avoid it and ensure safe driving.

[0122] In an optional manner, the first determination unit 1002 is specifically used to convert the coordinate values ​​of the radar coordinate system where the vehicle is located in the second data into coordinate values ​​of the world coordinate system through a first conversion matrix; the first conversion matrix is ​​determined based on the positional relationship between the radar device and the video image acquisition device; the coordinate values ​​of the world coordinate system are converted into coordinate values ​​of the coordinate system where the video image acquisition device is located through a second conversion matrix; the second conversion matrix is ​​different from the first conversion matrix; and a preset rule is used to convert the coordinate values ​​of the coordinate system where the video image acquisition device is located into coordinate values ​​of the pixel coordinate system.

[0123] The coordinate values ​​of the second data in the pixel coordinate system determined in this way are more accurate and reliable.

[0124] In an optional manner, the first determination unit 1002 is specifically used to perform imaging projection on the coordinate values ​​of the coordinate system where the video image acquisition device is located based on the focal length of the video image acquisition device to obtain the coordinate values ​​of the phase plane coordinate system of the video image acquisition device; based on the size of the pixels on the photosensitive chip of the video image acquisition device, the plane center of the phase plane coordinate system and the coordinate values ​​of the second data in the phase plane coordinate system, determine the coordinate values ​​of the second data in the pixel coordinate system after discretization.

[0125] The coordinate values ​​of the second data in the pixel coordinate system determined in this way are more accurate and reliable.

[0126] After introducing the method and apparatus for detecting abnormal lane change of a vehicle in an exemplary embodiment of the present application, next, a computing device in another exemplary embodiment of the present application is introduced.

[0127] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0128] In some possible implementations, the computing device according to the present application may include at least one processor and at least one memory. The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the vehicle abnormal lane change detection method according to various exemplary embodiments of the present application described above in this specification. For example, the processor may execute the following steps: Figure 2 Steps 201 to 203 shown in FIG.

[0129] Refer to the following Fig.11 hereinafter, a computing device 130 according to this embodiment of the present application is described. Fig.11 The computing device 130 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present application. Fig.11 As shown, the computing device 130 is in the form of a general intelligent terminal. The components of the computing device 130 may include but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0130] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus structures. Memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322, and may further include a read-only memory (ROM) 1323. Memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0131] The computing device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), and / or communicate with any device (e.g., routers, modems, etc.) that enables the computing device 130 to communicate with one or more other intelligent terminals. Such communication may be performed through an input / output (I / O) interface 135. In addition, the computing device 130 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 136. As shown, the network adapter 136 communicates with other modules for the computing device 130 through a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0132] In some possible implementations, various aspects of the method for detecting abnormal lane changes of a vehicle provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to cause the computer device to execute the steps of the method for detecting abnormal lane changes of a vehicle according to various exemplary embodiments of the present application described above in this specification. For example, the processor may execute the following steps: Figure 2 Steps 201 to 203 shown in FIG.

[0133] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0134] The program product for detecting abnormal lane change of a vehicle in the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and may be run on a smart terminal. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0135] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which a readable computer program is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0136] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0137] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0138] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable access frequency prediction device to produce a machine, so that the instructions executed by the processor of the computer or other programmable access frequency prediction device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable access frequency prediction device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0142] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting abnormal lane change of a vehicle, characterized in that: include: For the first road monitoring area, first data collected by the video image acquisition device and second data collected by the radar device are respectively acquired; The number of vehicles in the second data is greater than or equal to the number of vehicles in the first data; the second data includes motion attribute information of the vehicles; Determining video attribute information of vehicles in the first road monitoring area according to the second data and the first data; determining an abnormal lane change of a vehicle in the first road monitoring area according to the video attribute information of the vehicle, the motion attribute information of the vehicle, and the lane information in the first road monitoring area; Wherein, determining the abnormal lane change situation of the vehicle in the first road monitoring area according to the video attribute information of the vehicle, the motion attribute information of the vehicle and the lane information in the first road monitoring area includes: Determining a first target vehicle within the visual range of the video image acquisition device according to the video attribute information of the vehicle; Determining whether a baseline of the first target vehicle intersects a solid line of the lane according to the motion attribute information of the first target vehicle and the lane information in the first road monitoring area; If they intersect, determining that the first target vehicle has changed lanes abnormally; The determining, according to the second data and the first data, the video attribute information of the vehicle in the first road monitoring area includes: Convert the coordinate values ​​of the coordinate system of the radar device where the vehicle is located in the second data into coordinate values ​​of the world coordinate system, coordinate values ​​of the coordinate system where the video image acquisition device is located, and coordinate values ​​of the pixel coordinate system in order; the pixel coordinate system is the coordinate system where the first data is located; the coordinate system where the video image acquisition device is located is the camera coordinate system; Performing video fusion of the second data and the first data to determine the number of vehicles in the first road monitoring area and the area where the missing vehicles in the first data are located; Based on deep learning, data processing is performed on the area where the missing vehicle in the first data is located to determine the color information, vehicle model information, and license plate information of the missing vehicle; The video attribute information of the vehicles in the first road monitoring area is determined based on the first data, the color information, the vehicle model information, and the license plate information of the missing vehicle; the video attribute information includes: color information, vehicle model information, license plate information, and quantity information.

2. The method according to claim 1, characterized in that The determining, based on the video attribute information of the vehicle, the motion attribute information of the vehicle and the lane information in the first road monitoring area, an abnormal lane change situation of the vehicle in the first road monitoring area further includes: Determining a second target vehicle within the visual range of the video image acquisition device according to the video attribute information of the vehicle; Determining the motion attribute information of the second target vehicle outside the sight range of the video image acquisition device; Predicting the first location area where the second target vehicle is located in the next radar cycle based on a Kalman filter algorithm; Acquire a second location area where the second target vehicle is located in the next radar cycle; If the first position area is different from the second position area, and the distance between the second position area and the lane solid line is less than the width threshold of the second target vehicle, it is determined that the second target vehicle has changed lanes abnormally.

3. The method according to claim 2, characterized in that Also includes: According to the video attribute information of the vehicle, the abnormal lane-changing vehicle is determined to be a third target vehicle; the third target vehicle is one or more of the following: a police car, an ambulance, a fire truck, and an engineering vehicle; Determine the latitude and longitude coordinates, vehicle type, and license plate information of the third target vehicle.

4. The method according to claim 1, characterized in that The step of converting the coordinate values ​​of the coordinate system of the radar device where the vehicle is located in the second data into coordinate values ​​of the world coordinate system, coordinate values ​​of the coordinate system where the video image acquisition device is located, and coordinate values ​​of the pixel coordinate system in order includes: The coordinate values ​​of the radar coordinate system where the vehicle is located in the second data are converted into coordinate values ​​of the world coordinate system by using a first conversion matrix; the first conversion matrix is ​​determined based on the positional relationship between the radar device and the video image acquisition device; The coordinate values ​​of the world coordinate system are converted into the coordinate values ​​of the coordinate system where the video image acquisition device is located by a second conversion matrix; the second conversion matrix is ​​different from the first conversion matrix; The coordinate values ​​of the coordinate system where the video image acquisition device is located are converted into coordinate values ​​of the pixel coordinate system using a preset rule.

5. The method according to claim 4, characterized in that The method of converting the coordinate values ​​of the coordinate system where the video image acquisition device is located into the coordinate values ​​of the pixel coordinate system based on discretization processing according to a preset rule includes: Performing imaging projection on the coordinate values ​​of the coordinate system where the video image acquisition device is located based on the focal length of the video image acquisition device to obtain the coordinate values ​​of the phase plane coordinate system of the video image acquisition device; Based on the size of pixels on the photosensitive chip of the video image acquisition device, the plane center of the phase plane coordinate system and the coordinate value of the second data in the phase plane coordinate system, the coordinate value of the second data in the pixel coordinate system after discretization is determined.

6. A detection device for abnormal lane change of a vehicle, characterized in that: include: An acquisition unit, used to acquire, for a first road monitoring area, first data acquired by a video image acquisition device and second data acquired by a radar device; The number of vehicles in the second data is greater than or equal to the number of vehicles in the first data; the second data includes motion attribute information of the vehicles; A first determining unit, configured to determine video attribute information of vehicles in the first road monitoring area according to the second data and the first data; a second determining unit, configured to determine an abnormal lane change of a vehicle in the first road monitoring area according to the video attribute information of the vehicle, the motion attribute information of the vehicle, and the lane information in the first road monitoring area; The second determination unit is specifically used to determine the first target vehicle within the visual range of the video image acquisition device according to the video attribute information of the vehicle; determine whether the baseline of the first target vehicle intersects with the solid line of the lane according to the motion attribute information of the first target vehicle and the lane information in the first road monitoring area; if they intersect, determine that the first target vehicle has abnormally changed lanes; The second determination unit is specifically used to convert the coordinate values ​​of the coordinate system of the radar device where the vehicle is located in the second data into coordinate values ​​of the world coordinate system, the coordinate values ​​of the coordinate system where the video image acquisition device is located, and the coordinate values ​​of the pixel coordinate system in sequence; the pixel coordinate system is the coordinate system where the first data is located; the coordinate system where the video image acquisition device is located is the camera coordinate system; the second data is video-fused with the first data to determine the number of vehicles in the first road monitoring area and the area where the missing vehicles in the first data are located; data processing is performed on the area where the missing vehicles in the first data are located based on deep learning to determine the color information, vehicle model information, and license plate information of the missing vehicles; based on the first data and the color information, vehicle model information, and license plate information of the missing vehicles, the video attribute information of the vehicles in the first road monitoring area is determined; the video attribute information includes: color information, vehicle model information, license plate information, and quantity information.

7. A computing device, characterized in that: include: Memory and processor; A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 5 according to the obtained program.

8. A computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the method according to any one of claims 1-5.

Images