Method, device and terminal equipment for detecting violation behavior

Abstract

The application discloses a method, device and terminal equipment for detecting illegal behavior. The method comprises the following steps: determining a first area in an original image occupied by a target vehicle; determining a second area based on a target feature of the target vehicle, wherein the target feature is acquired in the first area, and the second area contains a target scene; and determining whether the target vehicle has illegal behavior according to a lane type contained in the target scene in the second area. The application solves the technical problem that the image requirement for a lane line image is high when a road monitoring camera is used to identify a lane type of a vehicle.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and terminal device for detecting violations. Background Technology

[0002] With socio-economic development and urban construction upgrades, people's travel needs are constantly increasing. In urban transportation, lane markings play an increasingly important role. There are various types of lanes, and different types of vehicles are allowed to travel in their designated lanes, and cannot arbitrarily occupy lanes where they are not permitted. For example, many cities have set up dedicated bus lanes to ensure efficient bus traffic; however, the act of private cars occupying bus lanes persists despite repeated prohibitions, especially during morning and evening rush hours. This not only violates traffic regulations but also seriously affects the operational efficiency of the public transportation system.

[0003] Traditional methods for detecting bus lane obstruction rely primarily on video footage captured by surveillance cameras, followed by manual screening. This approach is not only inefficient but also extremely costly. While artificial intelligence (AI) methods can detect lane obstruction, they require the identification of different lanes. However, current technologies are largely limited to surveillance cameras, with coverage constrained by the number and location of these cameras. Furthermore, lane identification based on camera images requires complete lane line images, placing high demands on the quality of the lane images and hindering its application in more flexible scenarios.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, and terminal device for detecting traffic violations, at least addressing the technical problem of high image requirements for lane line images when using road surveillance cameras to identify the lane type of a vehicle.

[0006] According to one aspect of the present invention, a method, apparatus, and terminal device for detecting traffic violations are provided, comprising: determining a first region in an original image occupied by a target vehicle; determining a second region based on target features of the target vehicle, wherein the target features are located and acquired within the first region, and the second region contains a target scene; and determining whether the target vehicle has committed a traffic violation based on the lane category contained in the target scene in the second region.

[0007] Optionally, determining the first region in the original image occupied by the target vehicle includes: performing target detection on the original image through a first neural network; if the detection result contains the target vehicle, then outputting the first region in the original image occupied by the target vehicle through the first neural network; the method further includes: if the detection result does not contain the target vehicle, then determining that the target vehicle does not have any traffic violations.

[0008] Optionally, determining the second region based on the target features of the target vehicle includes: locating the target features in the first region to determine the feature positions and auxiliary spacing; and determining the second region based on the feature positions and the auxiliary spacing.

[0009] Optionally, locating the target features in the first region and determining the feature positions and auxiliary spacing includes: locating the feature positions in the first region using a second neural network, wherein the number of outputs of the fully connected layers of the second neural network is the same as the number of target features; using the distance between the feature positions as the auxiliary spacing, or using the width of the target vehicle and / or the height of the target vehicle as the auxiliary spacing.

[0010] Optionally, determining the second region based on the feature location and the auxiliary spacing includes: determining the center position of the target feature based on the feature location; using the center position as the position parameter of the second region and the auxiliary spacing as the size parameter of the second region to determine the second region.

[0011] Optionally, determining the center position of the target feature based on the feature position includes: when the original image contains multiple target features, selecting the point closest to the target scene from the center positions corresponding to each target feature as the center position of the target feature.

[0012] Optionally, the target features include at least one of the following: wheels, license plates, headlights, and vehicle logos.

[0013] Optionally, after determining the second region based on the target features of the target vehicle, the method further includes: predicting the object-image distance of the target vehicle based on the original image; and scaling the second region proportionally according to the object-image distance.

[0014] Optionally, before determining the second region based on the target features of the target vehicle, the method further includes: performing attribute recognition on the target vehicle; if the target vehicle belongs to a preset category, then determining that the target vehicle has no traffic violations; if the target vehicle does not belong to the preset category, then determining the second region based on the target features of the target vehicle.

[0015] Optionally, the step of identifying the attributes of the target vehicle includes: identifying the attributes of the target vehicle through a third neural network; before determining whether the target vehicle has committed a traffic violation based on the lane category contained in the target scene in the second region, the step further includes: identifying the lane category contained in the corresponding second region in the original image through a fourth neural network.

[0016] Optionally, the third neural network and the fourth neural network add downsampling layers between convolutional layer groups for downsampling, and add short connections between convolutional layer groups.

[0017] According to another aspect of the present invention, a device for detecting traffic violations is also provided, comprising: an acquisition module for determining a first region occupied by a target vehicle in an original image; a determination module for determining a second region by combining the feature position of the target vehicle, wherein the feature position is located and acquired within the first region, and the second region contains a target scene; and a judgment module for determining whether the target vehicle has committed a traffic violation based on the lane category contained in the target scene in the second region.

[0018] According to another aspect of the present invention, a terminal device is also provided, comprising: a camera, a data processing module, and a storage module, wherein the camera is used to capture an original image; the data processing module is connected to the camera and is used to acquire the original image from the camera, and further used to determine that a target vehicle occupies a first region in the original image, and to determine a second region based on target features of the target vehicle, wherein the target features are located and acquired within the first region, and the second region contains a target scene; and to determine whether the target vehicle has committed a traffic violation based on the lane category contained in the target scene in the second region; the storage module is used to store the original image and the violation information of the target vehicle when the target vehicle illegally crosses a lane.

[0019] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is executed, the device where the non-volatile storage medium is located is controlled to perform the detection method for any of the above-described violations.

[0020] According to another aspect of the present invention, a computer device is also provided, the computer device including a processor, the processor being configured to run a program, wherein the program, when running, executes the detection method for any of the above-described violations.

[0021] In this embodiment of the invention, a lane recognition method is adopted by combining the image of the inspected vehicle and the lane image. The method involves determining that the target vehicle occupies a first region in the original image; determining a second region based on the target features of the target vehicle, wherein the target features are located and acquired in the first region and the second region contains the target scene; and determining whether the target vehicle has committed a violation based on the lane type contained in the target scene in the second region. This achieves the technical effect of providing a solution for detecting the lane type of a vehicle in flexible scenarios, thereby solving the technical problem of high image requirements for lane line images when using road monitoring cameras to identify the lane type of a vehicle. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0023] Figure 1 A hardware structure block diagram of a computer terminal for implementing a method for detecting violations is shown.

[0024] Figure 2 This is a flowchart illustrating a method for detecting violations according to an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of a vehicle position detection algorithm provided according to an optional embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a wheel positioning algorithm provided according to an optional embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of a bus lane identification algorithm provided by an optional embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of a vehicle type detection algorithm provided according to an optional embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of the bus lane occupancy detection process provided by an optional embodiment of the present invention;

[0030] Figure 8 This is a schematic diagram of the adaptive cropping process for the target region provided by an optional embodiment of the present invention;

[0031] Figure 9 This is a structural block diagram of a lane recognition device provided according to an embodiment of the present invention;

[0032] Figure 10 This is a structural block diagram of a terminal device provided according to an embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] According to an embodiment of the present invention, a method for detecting violations is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for a method of detecting violations is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the lane recognition method in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the lane recognition method of the aforementioned application. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0040] In existing technologies, to determine the type of lane a vehicle is traveling in, and thus whether the vehicle has entered a lane where it is prohibited, a high-angle camera is used to capture a panoramic view of the lane from a high vantage point, which is then fed into an image recognition model for lane identification. This method requires segmentation of the bus lane area, thus placing high demands on the viewing angle and resolution of the input image. It also requires complete extraction of lane line information and is only suitable for surveillance camera scenarios.

[0041] Figure 2This is a flowchart illustrating a method for detecting traffic violations according to an embodiment of the present invention. The invention can be applied to a terminal camera on a mobile platform, which can be a motor vehicle, a non-motor vehicle, or an unmanned mobile platform, etc. Based on the target feature localization of a target object ahead, the road surface image of the target object is extracted, and the lane type is determined based on the road surface image, thereby realizing lane type determination based on a single frame image captured by the terminal camera of the mobile platform. Figure 2 As shown, taking the mobile platform as a vehicle as an example, the method includes the following steps:

[0042] Step S202: Determine the first region occupied by the target vehicle in the original image;

[0043] Step S204: Determine the second region based on the target features of the target vehicle, wherein the target features are located and acquired in the first region, and the second region contains the target scene;

[0044] Step S206: Determine whether the target vehicle has committed a traffic violation based on the lane categories included in the target scene in the second area.

[0045] This optional implementation differs from the monitoring scenario; it's a bus lane occupancy detection scheme based on vehicle-mounted cameras, eliminating the need for complete extraction of lane lines and bus lane area information. By integrating vehicle detection and key point localization technologies, it employs an adaptive method to crop the road surface image around the vehicle. Bus lane identification is then performed using this road surface image to determine whether a lane is designated for buses. Compared to current solutions relying on lane line detection or segmentation technologies, this approach has a wider range of applications and performs better in high-traffic congestion scenarios.

[0046] The following is a detailed explanation of each of the above implementation steps.

[0047] Step S202: Determine the first region in the original image occupied by the target vehicle.

[0048] Optionally, the original image can be acquired from a vehicle-mounted camera, which can be installed on the following vehicle. The original image is the image captured by the vehicle-mounted camera of the following vehicle, with the target vehicle being the preceding vehicle traveling on the road. Furthermore, this application does not limit the specific location of the vehicle-mounted camera on the following vehicle, but the acquired original image must include an image of the preceding vehicle traveling on the road.

[0049] Specifically, the first region contains the location of the target vehicle in the original image, and accurate division of the first region is a prerequisite for subsequent recognition and classification. Due to the timing and location of the acquisition, the original image may contain all or part of the target vehicle. When the target vehicle is unobstructed, the image contains the complete target vehicle, and the first region contains the complete target vehicle. When the target vehicle is obstructed or located at the edge of the image and not fully included, the complete target vehicle is deduced from the known portion of the target vehicle in the first region.

[0050] As an optional embodiment, determining the first region in the original image occupied by the target vehicle includes: performing target detection on the original image using a first neural network; if the detection result contains the target vehicle, then outputting the first region in the original image occupied by the target vehicle using the first neural network. Furthermore, the above-mentioned method for detecting traffic violations may also include the following step: if the detection result does not contain the target vehicle, then determining that the target vehicle has not committed a traffic violation.

[0051] Specifically, before identifying the first area where the target vehicle is located, target detection can be performed based on the acquired original image. If no vehicle is detected, it is assumed that there is no vehicle occupying the bus lane in the image; otherwise, proceed to the next step. Figure 3 This is a schematic diagram of a vehicle position detection algorithm provided by an optional embodiment of the present invention, such as... Figure 3 As shown, the technical solution for the vehicle detection location algorithm can be obtained using a first neural network, which can be a convolutional neural network with short connections and a detection head. The specific process can be followed as follows:

[0052] First, the original image is input into the first neural network. Feature extraction is performed through four convolutional layers. A downsampling layer is added between every two convolutional layers to downsample the feature map, reducing its size to obtain a larger receptive field. To improve the effectiveness of feature extraction, short connections are added between every two convolutional layers to fuse feature information from upper and lower layers. Next, a fully connected layer predicts the location of vehicle detection bounding boxes and the confidence score of each box. Finally, a non-maximum suppression layer filters the vehicle bounding boxes from the previous step based on their confidence scores, retaining only the detection boxes with the highest confidence scores in adjacent regions. If the vehicle detection algorithm does not detect the target vehicle, it is determined that the target vehicle has not committed a traffic violation, and the process terminates; otherwise, the identified vehicle is taken as the target vehicle, and the first neural network outputs the first region in the original image occupied by the target vehicle. This embodiment ensures that the first region of the target vehicle is identified, avoiding the identification of the wheels of other vehicles in the image, which could cause errors in subsequent recognition and matching processes.

[0053] Step S204: Determine a second region based on the target features of the target vehicle, wherein the target features are located and acquired in the first region, and the second region contains the target scene.

[0054] Specifically, after detecting the target vehicle in the original image, target feature localization can be performed on the detected target vehicle, such as determining the wheel position, i.e., the target wheel's location in the original image. Furthermore, based on the target vehicle's target feature location, the region containing the target vehicle and its surrounding scene can be obtained. This avoids the need for complete extraction of lane lines and bus lane information in existing technologies, thereby improving the accuracy of violation detection types.

[0055] As an optional embodiment, determining the second region based on the target features of the target vehicle includes: locating the target features in the first region, determining the feature positions and auxiliary spacing; and determining the second region based on the feature positions and auxiliary spacing.

[0056] As an alternative embodiment, the target features include at least one of the following: wheels, license plates, headlights, and vehicle logos.

[0057] As an optional embodiment, determining the second region based on the feature location and auxiliary spacing includes: determining the center position of the target feature based on the feature location; using the center position as the position parameter of the second region and the auxiliary spacing as the size parameter of the second region to determine the second region.

[0058] As an optional embodiment, locating target features in the first region and determining feature positions and auxiliary spacing includes: obtaining feature positions in the first region through a second neural network, wherein the number of outputs of the fully connected layers of the second neural network is the same as the number of target features; using the distance between feature positions as auxiliary spacing, or using the width and / or height of the target vehicle as auxiliary spacing.

[0059] This application determines the center (position parameter) of the second region by the feature position of the target vehicle. Since the distance between the target vehicle and the current vehicle will affect the imaging size of the target vehicle, and thus affect the final size of the second region, this application also introduces an auxiliary distance as the size parameter of the second region, thereby effectively solving the influence of vehicle distance on the size of the region and improving the accuracy of the second region division.

[0060] In practical applications, target vehicles have a variety of features, and this application does not limit them to a single type. Target features may include at least one of the following: wheels, license plates, headlights, and vehicle logos.

[0061] Specifically, this application uses a second neural network to locate and obtain feature positions within a first region. The second neural network is a target feature localization technique that can employ a multi-branch convolutional neural network. Since different target features contain different numbers of features (e.g., four wheels, one license plate and one logo, and two headlights), the number of outputs from the fully connected layers of the second neural network in this application is the same as the number of detected target features. Taking four wheels as an example... Figure 4 This is a schematic diagram of a wheel alignment algorithm provided by an optional embodiment of the present invention, such as... Figure 4 As shown, in this method, the second neural network simultaneously predicts the positions of the four wheels using four parallel fully connected layers. If the wheels of the target vehicle are not visible in the original image, a default value is output. Even if the target vehicle is occluded and not all target features can be detected, the feature location can still be obtained when any target feature is detected.

[0062] As an optional embodiment, determining the center position of a target feature based on its feature location includes: when the original image contains multiple target features, selecting the point closest to the target scene from the center positions corresponding to each target feature as the center position of the target feature.

[0063] Furthermore, this application requires determining the center position as the location parameter of the region. When only one target feature is included, the midpoint of the feature location is taken as the center position. It should be noted that if there is only one target wheel in the original image, the position of that target wheel is the center position; if there are two target wheels in the original image, the midpoint of the line connecting the two target wheels in the original image can be determined as the center position; if there is only one license plate in the original image, the center point of the license plate is taken as the center position. When the detection result includes multiple target features, each target feature corresponds to a center position. The point closest to the target scene is selected from multiple center positions as the center position of the target feature. Since the target scene is road information, the point closest to the target scene is the point closest to the ground. Therefore, the closer the point is to the target scene, the higher its priority. The priority order is: wheel, license plate, headlights, and vehicle logo. For example, when two headlights and two wheels are detected, the positions of the two wheels correspond to a center position A, and the positions of the two headlights correspond to a center position B. Since the wheels have a higher priority than the headlights, the center position A is selected as the position parameter for the second region.

[0064] Furthermore, this application uses the distance between feature locations as an auxiliary spacing, or the width and / or height of the target vehicle as an auxiliary spacing. The actual distance between target features is constant (e.g., the physical distance between vehicle headlights). Because changes in vehicle distance cause variations in the final feature spacing in the image, the feature spacing is inversely proportional to the vehicle distance; the smaller the vehicle distance, the larger the target vehicle image, thus requiring a larger feature spacing. To include more comprehensive target scene information, a larger area is needed, hence the larger the area division size parameter. By obtaining accurate area size parameters through auxiliary spacing, and adaptively adjusting the area range based on vehicle distance and target vehicle objects, the accuracy of violation detection is improved.

[0065] Specifically, the positional spacing between similar target features can be used as an auxiliary spacing. For example, when the target feature is a wheel, the spacing between wheels can be used as an auxiliary spacing. Similarly, the positional spacing between different types of target features can also be used as an auxiliary spacing, such as the spacing between a wheel and headlights, or the spacing between a license plate and a vehicle logo. Furthermore, the attributes of the target vehicle (vehicle height or width) can be used as an auxiliary spacing. Moreover, this application does not limit the combination of center position and auxiliary spacing. When the center point corresponding to the wheel position is selected as the center position, the auxiliary spacing can be the vehicle width, the spacing between headlights, or any spacing between wheels and headlights. Likewise, the diverse selection of auxiliary spacing allows the size parameters of the area to be determined by selecting any auxiliary spacing even when the target vehicle is occluded. This application does not limit the shape of the second area; it can be rectangular, circular, or elliptical, etc., but the second area needs to contain information about the target object and the target scene. This application determines the center position of the region and defines the range of the target region based on auxiliary spacing. This can ensure that the lanes in the traffic road image within the defined target region are the lanes used by the target vehicle, and avoid selecting lanes that are not related to the target vehicle, thereby improving the accuracy of lane type detection.

[0066] As an optional implementation, determining the second region based on the target feature localization result allows for adaptive cropping of the road surface image surrounding the target vehicle. Taking the wheels as the target feature as an example, the specific process for determining the second region is as follows: Determine the x-axis coordinate xl of the leftmost visible wheel of the target vehicle in the original image; similarly, determine the x-axis coordinate xr of the rightmost visible wheel; and the y-axis coordinates yt and yb of the top and bottom visible wheels. Then, calculate the center position of the wheel, i.e., the coordinates (cx, cy), where cx = (xl + xr) / 2 and cy = (yt + yb) / 2. Next, calculate the auxiliary spacing. First, determine the width of the wheel region w = xr - xl and h = yb - yt. Take the maximum value of w and h, which is the auxiliary spacing L = max(w, h). Finally, starting from the center point, expand by L / 2 pixels upwards and downwards, and by L pixels to the left and right, to obtain the cropped area of ​​the road surface surrounding the vehicle. This cropped area is the region of the target area.

[0067] As an optional embodiment, the range of the target region in the original image can also be determined as follows: based on the original image, predict the object-image distance of the target vehicle; and determine the range of the target region in the original image based on the object-image distance.

[0068] Those skilled in the art will understand that the object-to-image distance (O / I) refers to the distance between the object plane and the camera chip. In this invention, it refers to the distance between the target vehicle in the original image and the camera that captured the original image. Due to the imaging principle of near objects appearing larger and distant objects appearing smaller, the pixel range occupied by the target vehicle in the original image is related to its object-to-image distance. Therefore, in order to obtain the lane image of the target vehicle and minimize the acquisition of images of other lanes, the size of the target area can be determined based on the object-to-image distance of the target vehicle in the original image, thereby improving the rationality of the selection of the target area size. For example, the size of the target area can be reduced as the object-to-image distance of the target object increases, preferably ensuring that the lane lines on both sides of the lane at that object-to-image distance can be covered by the target area.

[0069] Step S206: Determine whether the target vehicle has committed a traffic violation based on the lane categories included in the target scene in the second area.

[0070] As an optional implementation, the lane category contained in the corresponding second region in the original image is identified by a fourth neural network.

[0071] Based on the image portion corresponding to the second region in the original image, the lane type of the traffic surface is identified. Optionally, the target region of the original image can be cropped, and the image portion corresponding to the cropped target region can be fed separately to the subsequent functional module (third neural network) for lane type identification and detection. Optionally, lane types can include ordinary motor vehicle lanes, speed-limited motor vehicle lanes, bus lanes, bicycle lanes, etc., and the lane type identification result can determine which type of lane the target vehicle is traveling in.

[0072] As an optional embodiment, the first neural network, the second neural network, the third neural network and the fourth neural network all have downsampling layers added between the convolutional layer groups for downsampling, and short connections added between the convolutional layer groups. The first neural network also includes a non-maximum suppression layer.

[0073] Through the above steps, the type of lane the target vehicle is traveling in can be detected, thus achieving the technical effect of providing a solution for detecting the type of lane a vehicle is traveling in flexible scenarios. This solves the technical problem of high image requirements for lane line images when using road monitoring cameras to identify the type of lane a vehicle is traveling in.

[0074] Figure 5 This is a schematic diagram of a bus lane recognition algorithm provided by an optional embodiment of the present invention, such as... Figure 5 As shown, by identifying bus lanes, it can be determined whether the road surface where the target vehicle is traveling is a bus lane. The fourth neural network can be a convolutional neural network with short connections. The specific process is similar to the vehicle type recognition scheme (the third neural network), the difference being that the final fully connected layer of the bus lane identification algorithm outputs two things: the confidence score for determining whether it is a bus lane and the confidence score for not being a bus lane. If it is identified as a bus lane, it can be determined that there is a vehicle occupying the bus lane in the original image.

[0075] Based on the requirements of traffic laws and regulations, different vehicles have corresponding lane types that they can drive in. If a vehicle crosses a lane and enters a lane that is not allowed to drive in, it can be detected by the method provided in this optional embodiment, thereby realizing the automatic detection of vehicles illegally occupying lanes.

[0076] As an optional application scenario, the method provided in this optional embodiment can be used to detect non-preset vehicles that illegally enter a preset lane.

[0077] As an optional embodiment, before determining the second region based on the target features of the target vehicle in step S204, the method further includes step S203, which involves identifying the attributes of the target vehicle; if the target vehicle belongs to a preset category, it is determined that the target vehicle does not have any traffic violations; if the target vehicle does not belong to a preset category, the second region is determined based on the target features of the target vehicle.

[0078] As an optional embodiment, the target vehicle's attributes are identified using a third neural network.

[0079] Specifically, taking bus lane identification as an example, we can first predict whether the target vehicle is a bus based on the original image. If it is a bus, the detection ends. If the target vehicle is not a bus, we can further identify whether the lane it occupies is a bus lane (preset lane). If the lane type is a bus lane, we can determine that the matching result between the target vehicle and the lane type it occupies is a failure. Since the lane type it occupies is a bus lane, we can further determine that the target vehicle illegally crosses the lane and illegally occupies the bus lane.

[0080] Figure 6 This is a schematic diagram of a vehicle type detection algorithm provided by an optional embodiment of the present invention, such as... Figure 6 As shown, an image including the target vehicle (the first region corresponding to the original image) can be used as the input image to perform type recognition on the target vehicle and determine whether the vehicle is a bus. This application uses a third neural network to perform attribute recognition on the target vehicle. The technical solution for type recognition can use a convolutional neural network with short connections. The specific process is similar to the vehicle location detection algorithm (first neural network), except that the vehicle type detection algorithm removes the non-maximum suppression layer and directly predicts the vehicle category through a fully connected layer (e.g., predicting whether the vehicle type is a bus or a non-bus).

[0081] Figure 7 This is a schematic diagram of the bus lane occupancy detection process provided by an optional embodiment of the present invention. Figure 8 This is a schematic diagram of the adaptive cropping process for the target region provided by an optional embodiment of the present invention. Figure 7 As shown, taking the vehicle wheel as the target feature and the bus lane as the target scene as an example, the following steps can be used to detect whether there are vehicles illegally occupying the bus lane:

[0082] Step S1: Obtain the input image from the camera and input it into the vehicle detection unit;

[0083] Step S2: The vehicle detection unit determines whether there is a vehicle in the original image. If no vehicle is detected, proceed to step S8; if a vehicle is detected, proceed to step S3.

[0084] Step S3: Perform vehicle attribute recognition to determine whether the type of vehicle in the input image is a bus. If it is a bus, proceed to step S8; otherwise, proceed to step S4.

[0085] Step S4: Perform key point localization on the vehicle and identify the wheel coordinates of the vehicle.

[0086] Step S5: Adaptively crop the original input image to obtain a cropped image that includes the vehicle's wheels and the road surface on which the vehicle is traveling.

[0087] Step S6: Identify the lane type on the road surface in the cropped image. If it is identified as a non-bus lane, proceed to step S8; if it is identified as a bus lane, proceed to step S7.

[0088] Step S7: Determine if there are vehicles illegally occupying the bus lane in the input image;

[0089] Step S8: Determine that there are no vehicles illegally occupying the bus lane in the input image.

[0090] Figure 8 The demonstration focuses on the determination and cropping process of the target region based on entity images. During the key point localization of the vehicle, the wheels of the vehicle outlined in the image were located, and the subsequent cropping area was determined based on the position of the wheels. The cropping area is the aforementioned target region.

[0091] This optional implementation differs from the monitoring scenario; it's a bus lane occupancy detection scheme based on vehicle-mounted cameras, eliminating the need for complete extraction of lane lines and bus lane area information. By integrating vehicle detection and key point localization technologies, it employs an adaptive method to crop the road surface image around the vehicle. Bus lane identification is then performed using this road surface image to determine whether a lane is designated for buses. Compared to current solutions relying on lane line detection or segmentation technologies, this approach has a wider range of applications and performs better in high-traffic congestion scenarios.

[0092] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that the lane recognition method according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0094] According to embodiments of the present invention, a lane recognition device for implementing the above-described lane recognition method is also provided. Figure 9 This is a structural block diagram of a lane recognition device provided according to an embodiment of the present invention, such as... Figure 9 As shown, the lane recognition device includes: an acquisition module 92, a determination module 94, and a judgment module 96. The lane recognition device will be described below.

[0095] The acquisition module 92 is used to determine the first region occupied by the target vehicle in the original image;

[0096] The determination module 94 is used to determine the second region by combining the feature position of the target vehicle, wherein the feature position is located and obtained in the first region, and the second region contains the target scene;

[0097] The judgment module 96 is used to determine whether the target vehicle has committed a traffic violation based on the lane category included in the target scene in the second area.

[0098] It should be noted that the acquisition module 92, the determination module 94, and the judgment module 96 mentioned above correspond to steps S202 to S206 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in the embodiments.

[0099] According to embodiments of the present invention, a terminal device applicable to vehicles is also provided. Figure 10 This is a structural block diagram of a terminal device provided according to an embodiment of the present invention, such as... Figure 10 As shown, the terminal device includes: a camera 1002, a data processing module 1004, and a storage module 1006, wherein,

[0100] Camera 1002 is used to capture raw images;

[0101] The data processing module 1004 is connected to the camera 1002 and is used to acquire the original image from the camera 1002. It is also used to determine that the target vehicle occupies a first region in the original image and to determine a second region based on the target features of the target vehicle. The target features are located and acquired in the first region, and the second region contains the target scene. Based on the lane category contained in the target scene in the second region, it is determined whether the target vehicle has committed a traffic violation.

[0102] Storage module 1006 is used to store the original image and violation information of the target vehicle when the target vehicle illegally crosses the lane.

[0103] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0104] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the lane recognition method and device in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the lane recognition method described above. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: determining a first region in the original image occupied by the target vehicle; determining a second region based on the target features of the target vehicle, wherein the target features are located and acquired within the first region, and the second region contains the target scene; and determining whether the target vehicle has committed a traffic violation based on the lane category contained in the target scene within the second region.

[0106] Optionally, the processor may also execute program code that performs the following steps: determining a second region based on the target features of the target vehicle, including: locating the target features in the first region, determining the feature positions and auxiliary spacing; and determining the second region based on the feature positions and auxiliary spacing.

[0107] Optionally, the processor may also execute program code for the following steps: locating target features in the first region and determining feature positions and auxiliary spacing, including: locating and obtaining feature positions in the first region through a second neural network, wherein the number of outputs of the fully connected layers of the second neural network is the same as the number of target features; using the distance between feature positions as auxiliary spacing, or using the width and / or height of the target vehicle as auxiliary spacing.

[0108] Optionally, the processor may also execute program code that performs the following steps: determining a second region based on the feature location and auxiliary spacing, including: determining the center position of the target feature based on the feature location; using the center position as the position parameter of the second region and the auxiliary spacing as the size parameter of the second region, to determine the second region.

[0109] Optionally, the processor may also execute program code that performs the following steps: determining the center position of the target feature based on the feature position, including: when the original image contains multiple target features, selecting the point closest to the target scene from the center positions corresponding to each target feature as the center position of the target feature.

[0110] Optionally, the processor may also execute program code for the following steps: after determining the second region based on the target features of the target vehicle, the method further includes: predicting the object-image distance of the target vehicle based on the original image; scaling the second region proportionally according to the object-image distance.

[0111] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0112] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the lane recognition method provided in the above embodiments.

[0113] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0114] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring an original image, wherein the original image is an image of a target vehicle driving on a traffic surface; identifying the target position of the target wheel of the target vehicle in the original image; determining the area range of the target region in the original image based on the target position, wherein the area range of the target region includes the target position and the traffic surface; and identifying the lane type of the traffic surface based on the image portion in the original image corresponding to the target region.

[0115] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining the region range of the target area in the original image based on the target location, including: determining the center position of the target wheel in the original image based on the target location; and determining the region range included in the original image based on the center position, wherein the geometric center of the target area is the center position.

[0116] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining the range of the target region included in the original image based on the center position, including: in the case that the original image includes multiple wheels of the target vehicle, determining the maximum distance between the multiple wheels based on the target position and the center position of the multiple wheels; and determining the range of the target region included in the original image based on the maximum distance and the center position.

[0117] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining the range of the target region in the original image based on the center position, including: predicting the object-image distance of the target vehicle based on the original image; and determining the range of the target region in the original image based on the object-image distance.

[0118] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: identifying the target position of at least one wheel of the target vehicle in the original image, including: acquiring the original image; predicting a detection box corresponding to the target vehicle based on the original image; identifying the wheel of the target vehicle based on the detection box, and determining the target position of the wheel in the original image.

[0119] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: predicting the vehicle type of the target vehicle based on the original image; determining the matching result between the vehicle type of the target vehicle and the lane type of the traffic surface; and determining whether the target vehicle illegally crosses lanes based on the matching result.

[0120] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0121] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0126] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting traffic violations, characterized in that, include: Identify the first region in the original image occupied by the target vehicle; A second region is determined based on the target features of the target vehicle, wherein the target features are located and acquired within the first region, and the second region contains the target scene; Based on the lane categories included in the target scenario in the second region, determine whether the target vehicle has committed a traffic violation; The step of determining the second region based on the target features of the target vehicle includes: locating and obtaining feature positions within the first region using a second neural network, wherein the number of outputs of the fully connected layers of the second neural network is the same as the number of target features; using the distance between the feature positions as an auxiliary spacing, or using the width and / or height of the target vehicle as the auxiliary spacing; determining the center position of the target features based on the feature positions; and determining the second region using the center position as the position parameter of the second region and the auxiliary spacing as the size parameter of the second region.

2. The method according to claim 1, characterized in that, Determining that the target vehicle occupies a first region in the original image includes: Target detection is performed on the original image using a first neural network; If the detection result contains the target vehicle, then the first neural network outputs the first region in the original image occupied by the target vehicle; The method further includes: if the detection result does not include the target vehicle, then it is determined that the target vehicle has not committed any traffic violations.

3. The method according to claim 1, characterized in that, Determining the center position of the target feature based on the feature location includes: When the original image contains multiple target features, the point closest to the target scene is selected from the center positions corresponding to each target feature as the center position of the target feature.

4. The method according to claim 1, characterized in that, The target features include at least one of the following: wheels, license plates, headlights, and vehicle logos.

5. The method according to claim 1, characterized in that, After determining the second region based on the target features of the target vehicle, the method further includes: Based on the original image, predict the object distance of the target vehicle; The second region is scaled proportionally based on the object distance.

6. The method according to claim 1, characterized in that, Before determining the second region based on the target features of the target vehicle, the method further includes: Attribute identification is performed on the target vehicle; If the target vehicle belongs to a preset category, it is determined that the target vehicle has no traffic violations. If the target vehicle does not belong to the preset category, the second region is determined based on the target characteristics of the target vehicle.

7. The method according to claim 6, characterized in that, The attribute identification of the target vehicle includes: The target vehicle is attribute-identified using a third neural network. Before determining whether the target vehicle has committed a traffic violation based on the lane category included in the target scene in the second region, the method further includes: The lane category contained in the corresponding second region in the original image is identified by a fourth neural network.

8. The method according to claim 7, characterized in that, The third neural network and the fourth neural network each add downsampling layers between convolutional layer groups for downsampling, and add short connections between convolutional layer groups.

9. A device for detecting violations, characterized in that, include: The acquisition module is used to determine the first region occupied by the target vehicle in the original image; The determination module is used to determine a second region by combining the feature position of the target vehicle, wherein the feature position is located and obtained within the first region, and the second region contains the target scene; The judgment module is used to determine whether the target vehicle has committed a traffic violation based on the lane category included in the target scene in the second area; The determining module is further configured to locate and obtain feature positions within the first region using a second neural network, wherein the number of outputs of the fully connected layers of the second neural network is the same as the number of target features; use the distance between the feature positions as an auxiliary spacing, or use the width and / or height of the target vehicle as the auxiliary spacing; determine the center position of the target feature based on the feature positions; and determine the second region using the center position as the position parameter of the second region and the auxiliary spacing as the size parameter of the second region.

10. A terminal device, characterized in that, include: The camera, data processing module, and storage module, among which, The camera is used to capture raw images; The data processing module is connected to the camera and is used to acquire the original image from the camera. It is also used to determine that the target vehicle occupies a first region in the original image, and to determine a second region based on the target features of the target vehicle. The target features are located and acquired within the first region, and the second region contains the target scene. Based on the lane category contained in the target scene in the second region, it is determined whether the target vehicle has committed a traffic violation. The storage module is used to store the original image and the violation information of the target vehicle when the target vehicle illegally crosses the lane. The data processing module is further configured to locate and obtain feature positions within the first region using a second neural network, wherein the number of outputs of the fully connected layers of the second neural network is the same as the number of target features; use the distance between the feature positions as an auxiliary spacing, or use the width and / or height of the target vehicle as the auxiliary spacing; determine the center position of the target feature based on the feature positions; and determine the second region using the center position as the position parameter of the second region and the auxiliary spacing as the size parameter of the second region.

11. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the violation detection method according to any one of claims 1 to 8.

12. A computer device, characterized in that, The computer device includes a processor for running a program, wherein the program executes the method for detecting violations according to any one of claims 1 to 8.

Images