A method, apparatus and electronic device for determining an abnormal vehicle

Abstract

The application discloses a method, device and electronic equipment for determining an abnormal vehicle to avoid the low accuracy of the prior art in determining an abnormal vehicle, especially in a large traffic flow and / or large passenger flow scene. The method comprises: acquiring a first image set; wherein the first image set is composed of acquisition images collected for a monitoring area; determining a key vehicle with an open door state based on image features of vehicles in the first image set; determining a specified door with an open door state, and determining a target distance between a target object and the key vehicle; wherein the specified door is used for the target object to enter and / or exit the key vehicle, and the target object is detected in a relevant area of the acquisition image from the key vehicle; and determining whether the key vehicle is an abnormal vehicle based on the target distance.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus and electronic device for identifying abnormal vehicles. Background Technology

[0002] To ensure road traffic order and passenger / pedestrian safety, passengers must board / alight at designated locations set up by the transportation department. Correspondingly, vehicles are only permitted to stop at designated locations for passenger pick-up / drop-off. However, some drivers frequently park haphazardly to save time, which is extremely dangerous for passengers and easily causes traffic congestion. Although the transportation department invests significant manpower in monitoring key locations, oversights are inevitable, especially in temporary parking areas with high pedestrian and vehicle traffic. These areas only allow temporary parking and strictly prohibit passenger pick-up / drop-off. Examples include train station entrances. Therefore, current technology lacks a method to accurately monitor vehicle parking and prevent unauthorized passenger pick-up / drop-off. Summary of the Invention

[0003] This application provides a method, apparatus, and electronic device for identifying abnormal vehicles, in order to avoid the problem of inaccurate monitoring that easily occurs in areas with high traffic and pedestrian flow when identifying abnormal vehicles manually in the prior art.

[0004] Firstly, this application provides a method for identifying abnormal vehicles, including:

[0005] Obtain a first image set; wherein the first image set consists of images collected for the monitored area;

[0006] Based on the image features of vehicles in the first image set, key vehicles whose doors are open are identified.

[0007] The system determines that the door in the open state is a designated door, and determines the target distance between the target object and the key vehicle; wherein, the designated door is used for the target object to enter and / or leave the key vehicle, and the target object is detected from the key vehicle in the associated area of ​​the acquired image;

[0008] Based on the target distance, determine whether the key vehicle is an abnormal vehicle.

[0009] This application embodiment determines the status of vehicles entering the monitoring area, marks vehicles with open doors as key vehicles that may perform abnormal behavior; and while determining that the open door corresponds to a designated door for a passenger position in the key vehicle, it determines the target objects around the key vehicle and the target distance between the target objects and the key vehicle, and then tracks the changes in the target distance by acquiring images to determine that the target objects enter and / or leave the key vehicle through the designated door. Thus, by determining that the open door of the key vehicle is a designated door and the target distance is determined, the vehicle is accurately identified as an abnormal vehicle performing abnormal behavior.

[0010] One possible implementation, wherein determining whether the critical vehicle is an abnormal vehicle based on the target distance, includes:

[0011] In response to the target distance being less than a first preset threshold, the key vehicle is determined to be an abnormal vehicle.

[0012] One possible implementation includes determining that the door in the open state is a designated door, and determining the target distance between the target object and the key vehicle, comprising:

[0013] In response to the door being open being the designated door, the object type in the associated area is detected;

[0014] In response to the object type being the target object, the target distance between the target object and the key vehicle is determined.

[0015] One possible implementation, wherein determining that the door whose door state is open is the designated door, includes:

[0016] Based on the preset image coordinate system of the acquired image, a first key coordinate is determined; wherein, the first key coordinate indicates the position of the car door when the door is open;

[0017] Based on the correspondence between the image features of the vehicle and the vehicle orientation type, the orientation type of the key vehicle is determined; wherein, the vehicle orientation type includes a first orientation type and / or a second orientation type, and the vehicle orientation indicated by the first orientation type is different from the vehicle orientation indicated by the second orientation type;

[0018] Based on the first key coordinates, a first relative relationship is determined between the car door in the open state and the key vehicle; and based on the preset correspondence between the vehicle orientation type and the vehicle driving position, a second relative relationship is determined between the driving position of the key vehicle and the key vehicle.

[0019] Based on the first relative relationship and the second relative relationship, the relative relationship between the car door in the open state and the driving position of the key vehicle is determined;

[0020] In response to the relative relationship being opposite sides, the door whose state is open is determined to be the designated door.

[0021] One possible implementation, after determining the relative relationship between the open door and the driver's position of the key vehicle based on the first and second relative relationships, further includes:

[0022] In response to the relative relationship being on the same side, the rearview mirror features of the vehicle are acquired;

[0023] Based on the rearview mirror features of the vehicle and the image features of the first area corresponding to the door in the open state, it is determined whether the area where the door in the open state is located includes the rearview mirror; if yes, the door in the open state is determined to be the door corresponding to the driver's position; if no, the door in the open state is determined to be the designated door.

[0024] One possible implementation, before determining a key vehicle with an open door based on image features of vehicles in the first image set, further includes:

[0025] Based on the acquisition time of at least two acquired images in the first image set, a target vehicle with a driving speed less than a second preset threshold is determined.

[0026] Based on the acquisition time of the image of the target vehicle, determine the duration during which the driving speed of the target vehicle is less than the second preset threshold;

[0027] In response to the duration exceeding a third preset threshold, the target vehicle is determined to be a critical vehicle.

[0028] In one possible implementation, if the target object is a pedestrian, then the response after the target distance is less than a first preset threshold further includes:

[0029] In the first image set, images whose target distance is less than a second threshold are marked as the second image set; wherein, the second preset threshold is greater than the first preset threshold;

[0030] Based on the acquisition time of the images acquired in the second image set, pedestrians whose target distance is continuously less than the second preset threshold within a first preset time range are identified as target pedestrians;

[0031] Based on the image features of the second region corresponding to the key vehicle in the second image set, and the acquired luggage compartment features, the luggage compartment status of the key vehicle is determined.

[0032] In response to the key vehicle's luggage compartment being open, a third region image feature corresponding to the target pedestrian is determined in the second image set;

[0033] Based on the image features of the third region and the acquired baggage features, it is determined whether the third region where the target pedestrian is located includes baggage. If so, the target pedestrian is determined to be a passenger.

[0034] Secondly, embodiments of this application also provide a device for identifying abnormal vehicles, comprising:

[0035] An image unit is used to acquire a first image set, wherein the first image set consists of images acquired for a monitored area.

[0036] Vehicle unit: used to determine the key vehicle whose door status is open based on the image features of the vehicles in the first image set;

[0037] Distance unit: used to determine that the door in the open state is a designated door, and to determine the target distance between the target object and the key vehicle; wherein, the designated door is used for the target object to enter and / or leave the key vehicle, and the target object is detected from the key vehicle in the associated area of ​​the acquired image;

[0038] Anomaly Unit: Used to determine that the critical vehicle is an abnormal vehicle based on the target distance.

[0039] In one possible implementation, the anomaly unit is specifically used to determine that the critical vehicle is an abnormal vehicle in response to the target distance being less than a first preset threshold.

[0040] In one possible implementation, the distance unit is specifically used to detect the object type in the associated area in response to the door being open and the door being a designated door; and to determine the target distance between the target object and the key vehicle in response to the object type being the target object.

[0041] In one possible implementation, the distance unit is further configured to: determine first key coordinates based on a preset image coordinate system of the acquired image; wherein the first key coordinates indicate the position of the door that is open; determine the orientation type of the key vehicle based on the correspondence between the image features of the vehicle and the vehicle orientation type; wherein the vehicle orientation type includes a first orientation type and / or a second orientation type, the vehicle orientation indicated by the first orientation type being different from the vehicle orientation indicated by the second orientation type; determine a first relative relationship between the door that is open and the key vehicle based on the first key coordinates; and determine a second relative relationship between the driving position of the key vehicle and the key vehicle based on a preset correspondence between the vehicle orientation type and the vehicle driving position; determine the relative relationship between the door that is open and the driving position of the key vehicle based on the first and second relative relationships; and determine the door that is open as a designated door in response to the relative relationship being on opposite sides.

[0042] In one possible implementation, the distance unit is further configured to, in response to the relative relationship being on the same side, acquire the rearview mirror features of the vehicle; based on the rearview mirror features of the vehicle and the image features of a first region corresponding to the door in the open state, determine whether the region where the door in the open state is located includes the rearview mirror; if yes, then determine that the door in the open state is the door corresponding to the driver's position; if no, then determine that the door in the open state is the designated door.

[0043] In one possible implementation, the device further includes a rate unit, specifically configured to: determine a target vehicle whose driving speed is less than a second preset threshold based on the acquisition time of at least two acquired images in the first image set; determine the duration for which the driving speed of the target vehicle is less than the second preset threshold based on the acquisition time of the acquired image containing the target vehicle; and determine the target vehicle as a critical vehicle in response to the duration being greater than a third preset threshold.

[0044] In one possible implementation, where the target object is a pedestrian, the device further includes a baggage unit. Specifically, the baggage unit is used to mark images acquired in the first image set where the target distance is less than a second threshold as part of a second image set; wherein the second preset threshold is greater than the first preset threshold; based on the acquisition time of the images acquired in the second image set, determine pedestrians whose target distance is consistently less than the second preset threshold within a first preset time range as target pedestrians; based on the second region image features corresponding to a key vehicle in the second image set, and the acquired baggage compartment features, determine the baggage compartment status of the key vehicle; in response to the key vehicle's baggage compartment being open, determine the third region image features corresponding to the target pedestrian in the second image set; based on the third region image features and the acquired baggage features, determine whether the third region where the target pedestrian is located includes baggage; if so, determine that the target pedestrian is a passenger.

[0045] Thirdly, embodiments of this application also provide a readable storage medium, including,

[0046] memory,

[0047] The memory is used to store instructions that, when executed by a processor, cause an apparatus including the readable storage medium to perform the method as described in the first aspect and any possible implementation.

[0048] Fourthly, embodiments of this application also provide an electronic device, comprising:

[0049] Memory, used to store computer programs;

[0050] When a processor executes a computer program stored in the memory, it implements the method as described in the first aspect and any possible implementation. Attached Figure Description

[0051] Figure 1 A flowchart illustrating a method for determining abnormal vehicles provided in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of a vehicle orientation type in the acquired images provided in the embodiments of this application;

[0053] Figure 3 A schematic diagram of the target distance provided for embodiments of this application;

[0054] Figure 4 A schematic diagram illustrating the identification of abnormal vehicles based on acquired images, provided as an embodiment of this application;

[0055] Figure 5This is a schematic diagram illustrating how to determine the driving status of a vehicle based on acquired images, as provided in an embodiment of this application.

[0056] Figure 6 A schematic diagram of the structure of a device for determining abnormal vehicles provided in an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of the structure of an electronic device for identifying abnormal vehicles, provided in an embodiment of this application. Detailed Implementation

[0058] To address the lack of an accurate method for identifying abnormal vehicles in existing technologies, this application proposes a method for identifying abnormal vehicles: by identifying vehicles in a first set of acquired images, a key vehicle with an open door is identified; the open door of the key vehicle is identified as a designated door corresponding to a passenger position within the key vehicle; simultaneously, pedestrians in the acquired images are identified, and the pedestrian is determined to be a passenger of the key vehicle based on the target distance between the pedestrian and the passenger position of the vehicle, i.e., the pedestrian is determined to have entered and / or left the key vehicle through the designated door, thereby accurately identifying the key vehicle as an abnormal vehicle that has illegally picked up / dropped off passengers.

[0059] It should be noted that the acquired images or image sets composed of acquired images mentioned in the embodiments of this application are obtained by image acquisition devices for key monitoring areas. These image acquisition devices include, but are not limited to, surveillance cameras and cameras. Furthermore, the installation position (height, angle) of the image acquisition device should be such that it can capture the entire key monitoring area and clearly show the vehicle characteristics of vehicles within the monitoring area. Therefore, for any monitoring scenario with an area not exceeding a preset threshold, such as airport terminal entrances where only temporary parking is allowed and passenger pick-up / drop-off is prohibited, the method provided in the embodiments of this application only requires one image acquisition device to complete the monitoring of abnormal vehicle behavior.

[0060] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0061] Please refer to Figure 1 This application proposes a method for identifying abnormal vehicles to accurately determine the illegal passenger pick-up / drop-off behavior of vehicles when they stop. This method can be used to detect abnormal vehicles in road sections where passenger pick-up / drop-off is prohibited, and specifically includes the following implementation steps:

[0062] Step 101: Obtain the first image set.

[0063] The first image set consists of images collected for the monitored area, and therefore the images in the first image set can be monitoring screens corresponding to the monitored area.

[0064] It should be noted that the method provided in this application embodiment can be implemented based on a deep learning model. Therefore, after the image acquisition device acquires a first image set, the first image set can be input into the corresponding deep learning model. Accordingly, after acquiring the first image set, the deep learning model sequentially detects the acquired images in the first image set, thereby identifying key vehicles among multiple vehicles in the acquired images, and further identifying abnormal vehicles among the key vehicles based on the relationship between the key vehicles and the target pedestrian. That is, through the deep learning model, the acquired images in the first image set are determined, thereby identifying key vehicles and determining abnormal vehicles among the key vehicles.

[0065] Step 102: Based on the image features of vehicles in the first image set, determine the key vehicles whose doors are open.

[0066] The image features of a vehicle may include the vehicle's outline features when any of its doors is open. To accurately identify the door status, the image features may also include the vehicle's texture features corresponding to any door being open, as well as the area ratio between the open door and the vehicle body.

[0067] Furthermore, in this step of identifying key vehicles, it is first necessary to identify the vehicles in each acquired image in the first image set one by one, and then determine whether each vehicle is a key vehicle based on the image features of each vehicle.

[0068] Therefore, when the number of acquired images in the first image set is greater than or equal to 2, after determining the key vehicle in each acquired image, it is also necessary to determine the correspondence between key vehicles in different acquired images. This avoids the problem of decreased determination efficiency caused by repeatedly determining / detecting the same key vehicle in different acquired images due to the lack of a correspondence between key vehicles in different acquired images. Specifically, to determine the same key vehicle in multiple acquired images, a dynamic model can be established for the key vehicle in the acquired image. Based on this dynamic model, the position and instantaneous velocity of the aforementioned key vehicle in the reference acquired image corresponding to the acquired image can be predicted, and the determination can be made in the reference image based on the predicted information (position, instantaneous velocity). The determination of the same key vehicle can also be made using methods such as Kalman filtering.

[0069] Furthermore, to accurately identify key vehicles, verification can be performed by determining the vehicle's speed in the acquired images. Specifically, based on the acquisition times of at least two acquired images in the first image set, the speed of the vehicles in the acquired images is determined, and target vehicles with speeds less than a second preset threshold are identified. Next, based on the acquisition time of the image containing the target vehicle, the duration for which the target vehicle's speed is less than the second preset threshold is determined. If the duration exceeds a third preset threshold, the target vehicle is determined to be parked, and this target vehicle is identified as a key vehicle. Otherwise, to avoid incorrect judgments, a "speed-critical vehicle not parked" marker needs to be added to the key vehicles identified in the above steps.

[0070] It is worth noting that the determination of the target vehicle and the determination of the target vehicle's travel time at the corresponding speed can be performed before or after the key vehicle is determined. Both can play a role in assisting in the determination and verification of the key vehicle, thereby improving accuracy.

[0071] Furthermore, to determine the abnormal behavior of a critical vehicle, i.e., whether the critical vehicle is picking up or dropping off passengers, the determination of open doors in this embodiment is mainly used to distinguish between doors corresponding to the driver's seat and passenger seats. Therefore, determining whether an open door is a designated door in this embodiment is essentially determining whether an open door corresponds to a passenger seat. Thus, this designated door is used by the target individual to enter and / or leave the critical vehicle as a passenger.

[0072] The following provides a detailed explanation of how to determine whether a door that is in an open state is a specific door:

[0073] First, while identifying key vehicles in the first image set based on their image features, the system also detects open doors (i.e., doors in an open state) within these key vehicles. Based on a preset image coordinate system, the system determines and records the first key coordinates indicating the location of the open door. Specifically, the first key coordinates can be the coordinates of the center point of the open door or the coordinates of its vertices. When representing an open door using the center point coordinates, the open door can also be represented by shape parameters. For example, it can be represented by (x, y, l, w), where x is the abscissa of the center point, y is the ordinate of the center point, l is the length of the area containing the open door, and w is the width of the area containing the open door.

[0074] Then, based on the correspondence between vehicle image features and vehicle orientation types, the orientation type of the key vehicle can be determined. The orientation type includes a first orientation type and / or a second orientation type, where the vehicle orientation indicated by the first orientation type differs from that indicated by the second orientation type. The vehicle orientation indicated by the first orientation type can be opposite to that indicated by the second orientation type. Figure 2 This is a schematic diagram illustrating the vehicle orientation type in the acquired images provided in the embodiments of this application. For example... Figure 2 As shown, Figure 2 The vehicles in part (a) have different orientation types than those in part (b). Assuming that the vehicles in part (a) have the first orientation type, then the vehicles in part (b) have the second orientation type.

[0075] Next, based on the first key coordinate, the first relative positional relationship between the open door and the key vehicle is determined. Specifically, the abscissa of the first key coordinate and the abscissa of the center point of the second region can be compared, thereby determining whether the open door is located to the left or right of the key vehicle, with the aforementioned center point coordinate as a reference. The second region refers to the area where the key vehicle is located.

[0076] Simultaneously, based on the preset correspondence between vehicle orientation type and vehicle driving position, the driving position of key vehicles and the second relative relationship of key vehicles are determined. For details, please refer to... Figure 2 Based on the aforementioned pre-defined correspondence, the driving position is determined, for example, Figure 2 (a) If some vehicles are of the first orientation type, the driving position corresponding to the first orientation type should be located away from the image acquisition device and parallel to the currently observed rearview mirror, i.e., the area shown by the dashed line in part (a). It is worth noting that when all vehicles in the acquired image have only one orientation type due to driving on a one-way street, the orientation type is not determined here, but the first relative relationship and the second relative relationship are determined directly after the vehicle is detected.

[0077] Next, based on the first and second relative relationships, the relative relationship between the open door and the driver's position of the key vehicle is determined. For example, if the first relative relationship determines that the open door is located on the left side of the key vehicle, and the second relative relationship also determines that the driver's position is on the left side of the key vehicle, then when the first and second relative relationships are consistent and both use the key vehicle as a reference, it can be determined that the open door and the driver's position are on the same side. Similarly, if the first and second relative relationships are inconsistent, then when both use the key vehicle as a reference, it can be determined that the open door and the driver's position are on opposite sides.

[0078] In another embodiment of this application, a second key coordinate relative to the second area where the key vehicle is located can be determined based on the second relative relationship. Therefore, the first key coordinate and the second key coordinate can be compared. If the difference between the horizontal coordinates of the first key coordinate and the second key coordinate exceeds the same-side distance threshold, it is determined that the door with the door in the open state and the driver's position are located on opposite sides of the key vehicle, and the door with the door in the open state is determined to be the designated door. Conversely, if the difference between the horizontal coordinates of the first key coordinate and the second key coordinate does not exceed the same-side distance threshold, it is determined that the driver's position and the door with the door in the open state are on the same side. Therefore, the position of the door with the door in the open state may not be the designated door (i.e., the door corresponding to the driver's position), or it may be the designated door in the key vehicle corresponding to the passenger position behind the driver's position.

[0079] Next, in response to the opposite relationship, the door with the door status of being open is determined to be the designated door, that is, the door is used for passengers to enter and / or leave the key vehicle.

[0080] In response to the condition that the relative relationship is on the same side, it is necessary to further determine whether the door with the open state is the designated door, that is, whether the aforementioned door with the open state is the door in the key vehicle relative to the passenger position. Although it is relatively rare for passengers to get on / off through the door on the same side as the driver's seat due to the driver's habit of parking on the side of the road, for the sake of accuracy, further detection is performed on the first area feature where the door with the open state is located to determine the position of the door with the open state.

[0081] When the open door corresponds to the driver's position, the area (or detection frame) of the open door also includes the rearview mirror. However, when the open door corresponds to a specific door behind the driver's side passenger, due to blind spots, the area of ​​the open door in the captured image does not include the rearview mirror. This is particularly relevant when a critical vehicle is facing the image acquisition device from the rear, for example... Figure 2 The vehicle orientation type is shown in section (b). Therefore, in one embodiment of this application, in response to the relative relationship between an open door and the driver's position of the key vehicle being on the same side, the rearview mirror features of the vehicle are acquired; wherein, the rearview mirror features include the shape features of the rearview mirror and the relative positional relationship between the rearview mirror and the vehicle. Next, based on the rearview mirror features of the vehicle and the first area image features corresponding to the open door, that is, comparing the acquired rearview mirror features of the vehicle with the first area image features, it is determined whether the area where the open door is located includes a rearview mirror. If yes, the open door is determined to be the door corresponding to the driver's position. If no, the open door is determined to be the designated door corresponding to the passenger position behind the driver's position of the key vehicle.

[0082] Step 102 can be implemented based on a door state detection model. Therefore, this door state detection model includes the mapping relationship between vehicle image features and different door states, the correspondence between vehicle image features and vehicle orientation types, and a preset correspondence between vehicle orientation types and vehicle driving positions. This mapping relationship can be trained based on a labeled sample training set. Therefore, all the regions mentioned above, such as the first region or the second region, can correspond to detection boxes in the door state model.

[0083] Based on this, the vehicle door state detection model can classify vehicles with closed doors and key vehicles with open doors in the captured images during image detection, and output the first key coordinate of the open door state. Simultaneously, it can also classify the vehicle's orientation type into first and second orientation types to determine the corresponding vehicle's orientation type.

[0084] Step 103: Determine that the door in the open state is the designated door, and determine the target distance between the target object and the key vehicle.

[0085] Specifically, designated vehicle doors are used for target objects to enter and / or exit the critical vehicle. Target objects are detected from the critical vehicle within the associated region of the acquired image.

[0086] Specifically, to improve the efficiency of identifying abnormal vehicles, in one embodiment of this application, in response to the door being open as a designated door, object type detection is performed in the associated area. The object type includes a target object. This target object can be a pedestrian. Further, in response to the object type being a target object, a target distance between the target object and the key vehicle is determined. This target distance can be determined based on the third key coordinate of the key vehicle and the fourth key coordinate of the pedestrian.

[0087] The third and fourth key coordinates can be represented by the center coordinates of the area where the key vehicle is located and the center coordinates of the area where the pedestrian is located in the preset image coordinate system, respectively. Figure 3 This is a schematic diagram illustrating the relative position between a key vehicle and a target object, provided as an embodiment of this application. Figure 3 As shown, area A is the location of the critical vehicle, and area B is the location of the target object (e.g., a pedestrian). The coordinates of the center point of area A are shown. The coordinates of the center point of region B are: Among them, (X) max_v Y max_v ), (X max_v Y min_v(X) represents two vertices in the preset image coordinate system whose x and y coordinates are different in region A. max_p Y max_p ), (X max_p Y min_p Let A and B be two vertices in the preset image coordinate system whose x and y coordinates are different. Therefore, the target distance D can be determined by the following formula:

[0088]

[0089] It should be noted that the embodiments of this application do not limit the execution order of determining the specified car door and determining the target distance in step 103. Furthermore, step 103 can also be executed based on an object detection model in a deep learning model. Therefore, the object detection model includes the correspondence between the target object's movement features and the target object, and preset rules for determining the target distance based on the target object's key coordinates.

[0090] Step 104: Based on the target distance, determine whether the key vehicle is an abnormal vehicle.

[0091] Specifically, in response to a target distance greater than or equal to a first preset threshold, the key vehicle is determined to be a non-abnormal vehicle. In response to a target distance less than the first preset threshold, if it is determined that the pedestrian enters and / or leaves the key vehicle through the aforementioned designated door, then the pedestrian is a passenger of the key vehicle; that is, the key vehicle picks up / drops off passengers within the monitored area. Therefore, the key vehicle is an abnormal vehicle.

[0092] Furthermore, when the target object is a pedestrian getting on or off a vehicle, the trunk of the key vehicle is usually used to temporarily store luggage. Therefore, in one embodiment of this application, after the target distance is less than a first preset threshold, the system further determines whether the trunk of the key vehicle is open by using a pre-stored correspondence between the vehicle's trunk features and trunk status. For key vehicles with open trunks, pedestrians and luggage are detected. When pedestrians and luggage are detected, it can be further determined that the pedestrian is a passenger. This step can also be performed by a classification model that includes a pre-trained correspondence between trunk features and trunk status. Specifically, when the target distance is determined to be less than the first preset threshold, images with target distances less than a second preset threshold can be marked as the second image set in the first image set. The second preset threshold is greater than the first preset threshold. That is, when it is determined through the acquired images that the target distance of a pedestrian is less than the first threshold, the acquired image in which the pedestrian appears is selected, and the trajectory of the pedestrian after the target distance is less than the second preset threshold is tracked.

[0093] Meanwhile, if a pedestrian's destination is not related to the critical vehicle due to route planning errors, the pedestrian will quickly adjust even if the target distance is less than the second preset threshold. Therefore, after identifying pedestrians whose target distance is less than the second threshold, further filtering is performed on pedestrians who mistakenly enter the key monitoring area corresponding to the critical vehicle (target distance between the first and second preset thresholds) based on the image acquisition time: based on the image acquisition time in the second image set, pedestrians whose target distance is consistently less than the second preset threshold within the first preset time range are identified as target pedestrians. Next, the vehicle's luggage compartment features and luggage features are acquired. Based on the second region image features corresponding to the critical vehicle in at least one acquired image in the second image set, and the acquired luggage compartment features, the luggage compartment status of the critical vehicle is determined. The luggage compartment features include the relative positional relationship between the luggage compartment and the critical vehicle, and the contour features of the luggage compartment in different states, such as the contour features in the open state. In response to the key vehicle's luggage compartment being open, the system identifies the third region image features corresponding to the target pedestrian in the second image set, along with the acquired luggage compartment features, to determine whether luggage is present in that third region, thereby confirming that the target pedestrian is using the luggage compartment to store luggage. If so, it can be further determined that the target pedestrian is a passenger of the key vehicle. The luggage features include the outline features of at least one type of suitcase.

[0094] Furthermore, after determining that the vehicle is an abnormal vehicle, its vehicle information can be determined based on any of the captured images and uploaded. This vehicle information may include the license plate number.

[0095] Based on the method for identifying abnormal vehicles described in steps 101-104 above, the following examples illustrate this method. Please refer to them. Figure 4 .

[0096] In this example, the detection / identification of abnormal vehicles can be accomplished using a deep learning model. This deep learning model can specifically be a vehicle detection model, a door status detection model, a vehicle orientation classification model, and a pedestrian tracking model. After detecting that a door is open, the relative positional relationship between the driver's seat and the open door in the key vehicle is determined to identify whether the open door corresponds to the designated door of the passenger seat in the key vehicle. This relative positional relationship is closely related to the vehicle's orientation (type). Furthermore, it should be noted that the aforementioned vehicle orientation classification model is mainly applied to scenarios where the captured images are not one-way streets. When the captured images show a one-way street, it is not necessary to use an orientation model for identification; instead, the relative positional relationship between the driver's position and the vehicle can be preset directly in the door detection model based on the one-way street direction. Secondly, all of the above deep learning models can be trained under supervision using their respective sample training sets. Therefore, the above deep learning models include the correspondence between vehicle image features and door status, and the relative positional relationships between key components such as the rearview mirror, trunk, and driver's seat and the key vehicle.

[0097] Specifically, first, the captured images from the image set are sequentially input into the deep learning model. In practice, the captured images first enter the vehicle detection model for detection: while detecting vehicles in the captured images, the license plates on the vehicles are also detected and their information is read; then, based on the time the images were captured, the vehicle's driving status is determined. Please refer to [reference needed]. Figure 5 If a vehicle is detected entering the monitoring area but has not crossed the capture line, no further detection of the vehicle's specific status will be performed until the vehicle crosses the capture line. Once a vehicle is detected entering the monitoring area and crossing the capture line, its speed is detected based on the acquisition time of the captured image: if the vehicle speed gradually decreases until it enters a speed range, the driving status is determined to be "entering". Similarly, if the vehicle speed is within the entry speed range after a preset stopping time range, and then gradually increases until it leaves the monitoring area, the driving status is determined to be "exiting". Conversely, if the vehicle speed does not enter the first speed range and leaves the monitoring area within a preset passing time range, the driving status is determined to be "passing".

[0098] The method described in this application primarily detects vehicles in the driving states of leaving and / or entering. The difference in vehicle detection between these two driving states lies in the following: when an entering vehicle is detected, after feature extraction, subsequent images following the detected images are analyzed for door status / orientation, etc. That is, a door status detection model is used to detect subsequent images (frames), without analyzing / calculating previous images or the current image. For exiting vehicles, after extracting the corresponding vehicle features, previous historical images (frames) are detected and analyzed.

[0099] Next, in the images captured for vehicles in different driving states, the vehicle door state detection model is used to determine which vehicles have open doors. When the door is open (V_Door = 1), the vehicle's speed should be 0. Otherwise, vehicles with closed doors are marked as having closed doors (V_Door = 0).

[0100] Furthermore, for vehicles with open doors, the vehicle orientation classification model determines whether it is the designated door for passengers to enter / exit the vehicle. When it is determined to be the designated door, the presence of pedestrians around the vehicle is detected. If a pedestrian is detected, it is marked as P_Flage = 1; otherwise, it is marked as P_Flage = 0. Simultaneously, trajectory tracking is performed on all pedestrians. As the distance (P_V_D) between the detected pedestrian and the vehicle with the open door decreases, the vehicle's trunk status, i.e., whether the luggage compartment is open, is further classified. If it is (V_Trunk = 1), the luggage compartment is further detected (S_Flage = 1). If the luggage compartment is detected, it can be determined that the pedestrian is a passenger of the vehicle, and the passenger entered / exited the vehicle through the currently open door. Furthermore, the distance (P_V_D) between pedestrians and vehicles can be further detected (tracked) in the captured images where luggage is detected. If the distance increases, it means that the pedestrian opened the vehicle door, took out their luggage, and left the vehicle; correspondingly, the vehicle is disembarking within the monitored area of ​​the captured image. If the distance does not increase, it means that the pedestrian placed their luggage in the luggage compartment and boarded the vehicle through the open door; correspondingly, the vehicle is boarding within the monitored area of ​​the captured image.

[0101] Since passenger pick-up / drop-off is not permitted in the monitored area captured in the images, the vehicles and their corresponding license plate information can be marked as abnormal vehicles and uploaded. Furthermore, specific abnormal behaviors can also be marked and uploaded along with the images. For example, illegal passenger pick-up, suspected illegal passenger pick-up, or potentially suspected illegal passenger pick-up, etc. See the attached image for details. Figure 4 .

[0102] Based on the same inventive concept, this application provides a device for identifying abnormal vehicles, which is similar to the aforementioned device. Figure 1 The method for identifying abnormal vehicles is shown below. For a detailed description of the device, please refer to the foregoing description of the method embodiments; details will not be repeated here. Figure 6 The microcontroller in this device includes:

[0103] Image unit 601; used to acquire the first image set.

[0104] The first image set consists of images collected from the monitored area;

[0105] Vehicle unit 602: Used to determine a key vehicle whose door is open based on the image features of the vehicles in the first image set.

[0106] Distance unit 603: used to determine that a door with an open status is a designated door, and to determine the target distance between the target object and the key vehicle.

[0107] The designated door is used for the target object to enter and / or leave the key vehicle, and the target object is detected from the key vehicle in the associated area of ​​the acquired image.

[0108] The distance unit 603 is specifically used to detect the object type in the associated area in response to the door being open and the door being a specified door; and to determine the target distance between the target object and the key vehicle in response to the object type being the target object.

[0109] The distance unit 603 is further configured to: determine a first key coordinate based on a preset image coordinate system of the acquired image; wherein the first key coordinate indicates the position of the door that is open; determine the orientation type of the key vehicle based on the correspondence between the image features of the vehicle and the vehicle orientation type; wherein the vehicle orientation type includes a first orientation type and / or a second orientation type, the vehicle orientation indicated by the first orientation type being different from the vehicle orientation indicated by the second orientation type; determine a first relative relationship between the door that is open and the key vehicle based on the first key coordinate; and determine a second relative relationship between the driving position of the key vehicle and the key vehicle based on a preset correspondence between the vehicle orientation type and the vehicle driving position; determine the relative relationship between the door that is open and the driving position of the key vehicle based on the first and second relative relationships; and determine the door that is open as a designated door in response to the relative relationship being on opposite sides.

[0110] The distance unit 603 is further configured to, in response to the relative relationship being on the same side, acquire the rearview mirror features of the vehicle; based on the rearview mirror features of the vehicle and the first area image features corresponding to the door in the open state, determine whether the area where the door in the open state is located includes the rearview mirror; if yes, determine that the door in the open state is the door corresponding to the driver's position; if no, determine that the door in the open state is the designated door.

[0111] Anomaly Unit 604: Used to determine whether the critical vehicle is an abnormal vehicle based on the target distance.

[0112] The anomaly unit 604 is specifically used to determine that the key vehicle is an abnormal vehicle in response to the target distance being less than a first preset threshold.

[0113] The aforementioned device for identifying abnormal vehicles further includes a rate unit, which is specifically used to determine a target vehicle with a driving speed less than a second preset threshold based on the acquisition time of at least two acquired images in the first image set; to determine the duration for which the driving speed of the target vehicle is less than the second preset threshold based on the acquisition time of the acquired image of the target vehicle; and to determine that the target vehicle is a critical vehicle in response to the duration being greater than a third preset threshold.

[0114] If the target object is a pedestrian, the device for determining abnormal vehicles further includes a luggage unit. Specifically, the luggage unit is used to mark images acquired in the first image set where the target distance is less than a second threshold as part of a second image set; wherein the second preset threshold is greater than the first preset threshold; based on the acquisition time of the images acquired in the second image set, determine pedestrians whose target distance is consistently less than the second preset threshold within a first preset time range as target pedestrians; based on the second region image features corresponding to the key vehicle in the second image set, and the acquired luggage compartment features, determine the luggage compartment status of the key vehicle; in response to the luggage compartment status of the key vehicle being open, determine the third region image features corresponding to the target pedestrian in the second image set; based on the third region image features and the acquired luggage features, determine whether the third region where the target pedestrian is located includes luggage; if so, determine that the target pedestrian is a passenger.

[0115] Based on the same inventive concept, embodiments of this application also provide a readable storage medium, including:

[0116] memory,

[0117] The memory is used to store instructions that, when executed by a processor, cause the apparatus including the readable storage medium to perform the method for determining abnormal vehicles as described above.

[0118] Based on the same inventive concept, this application also provides an electronic device that can implement the aforementioned method for determining abnormal vehicles. Please refer to [link / reference]. Figure 7 The electronic device includes:

[0119] At least one processor 701 and a memory 702 connected to at least one processor 701. In this embodiment, the specific connection medium between the processor 701 and the memory 702 is not limited. Figure 7 The example shown is the connection between processor 701 and memory 702 via bus 700. Bus 700 is... Figure 7The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 700 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 7 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 701 can also be called a controller; there is no restriction on the name.

[0120] In this embodiment, memory 702 stores instructions executable by at least one processor 701. By executing the instructions stored in memory 702, at least one processor 701 can perform the method for determining abnormal vehicles described above. Processor 701 can implement... Figure 6 The functions of each module in the device shown.

[0121] The processor 701 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 702 and calling data stored in memory 702, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0122] In one possible design, processor 701 may include one or more processing units. Processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 701. In some embodiments, processor 701 and memory 702 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0123] The processor 701 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method for determining abnormal vehicles disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0124] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 702 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 702 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 702 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0125] By designing and programming the processor 701, the code corresponding to the method for determining abnormal vehicles described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the code during runtime. Figure 1 The steps of the method for identifying abnormal vehicles are shown. How to design and program the processor 701 is a technique well-known to those skilled in the art and will not be described further here.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0127] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: Universal Serial Bus flash disks, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0131] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for identifying abnormal vehicles, characterized in that, include: Obtain a first image set; wherein the first image set consists of images collected for the monitored area; Based on the image features of vehicles in the first image set, key vehicles whose doors are open are identified. Based on the preset image coordinate system of the acquired image, a first key coordinate is determined; wherein, the first key coordinate indicates the position of the car door when the door is open; Based on the correspondence between the image features of the vehicle and the vehicle orientation type, the orientation type of the key vehicle is determined; wherein, the vehicle orientation type includes a first orientation type and / or a second orientation type, and the vehicle orientation indicated by the first orientation type is different from the vehicle orientation indicated by the second orientation type; Based on the first key coordinates, a first relative relationship is determined between the car door in the open state and the key vehicle; and based on the preset correspondence between the vehicle orientation type and the vehicle driving position, a second relative relationship is determined between the driving position of the key vehicle and the key vehicle. Based on the first relative relationship and the second relative relationship, the relative relationship between the car door in the open state and the driving position of the key vehicle is determined; In response to the fact that the relative relationship between the open door and the driving position of the key vehicle is opposite, the open door is determined to be a designated door, and the target distance between the target object and the key vehicle is determined; wherein, the designated door is used for the target object to enter and / or leave the key vehicle, and the target object is detected from the key vehicle in the associated area of ​​the acquired image; Based on the target distance, determine whether the key vehicle is an abnormal vehicle.

2. The method as described in claim 1, characterized in that, The step of determining whether the key vehicle is an abnormal vehicle based on the target distance includes: In response to the target distance being less than a first preset threshold, the key vehicle is determined to be an abnormal vehicle.

3. The method as described in claim 1, characterized in that, The steps of determining that the door in the open state is the designated door, and determining the target distance between the target object and the key vehicle, include: In response to the door being open being the designated door, the object type in the associated area is detected; In response to the object type being the target object, the target distance between the target object and the key vehicle is determined.

4. The method as described in claim 1, characterized in that, After determining the relative relationship between the open door and the driver's position of the key vehicle based on the first and second relative relationships, the method further includes: In response to the fact that the relative relationship between the open door and the driving position of the key vehicle is on the same side, the rearview mirror features of the vehicle are obtained. Based on the rearview mirror features of the vehicle and the image features of the first area corresponding to the door that is open, it is determined whether the area where the door is open includes the rearview mirror; if yes, the door that is open is determined to be the door corresponding to the driver's position; if no, the door that is open is determined to be the designated door.

5. The method as described in claim 1, characterized in that, Before determining the key vehicle whose door status is open based on the image features of the vehicle in the first image set, the method further includes: Based on the acquisition time of at least two acquired images in the first image set, a target vehicle with a driving speed less than a second preset threshold is determined. Based on the acquisition time of the image of the target vehicle, determine the duration during which the driving speed of the target vehicle is less than the second preset threshold; In response to the duration exceeding a third preset threshold, the target vehicle is determined to be a critical vehicle.

6. The method as described in claim 2, characterized in that, If the target object is a pedestrian, then the response after the target distance is less than a first preset threshold further includes: In the first image set, images whose target distance is less than a second preset threshold are marked as the second image set; wherein, the second preset threshold is greater than the first preset threshold; Based on the acquisition time of the images acquired in the second image set, pedestrians whose target distance is continuously less than the second preset threshold within a first preset time range are identified as target pedestrians; Based on the image features of the second region corresponding to the key vehicle in the second image set, and the obtained luggage compartment features, the luggage compartment status of the key vehicle is determined. In response to the key vehicle's luggage compartment being open, in the second image set, determine The third region image features corresponding to the target pedestrian; Based on the image features of the third region and the acquired baggage features, it is determined whether the third region where the target pedestrian is located includes baggage. If so, the target pedestrian is determined to be a passenger.

7. A device for identifying abnormal vehicles, characterized in that, include: Image unit; Used to acquire a first image set; wherein the first image set consists of images acquired for the monitored area; Vehicle unit: used to determine the key vehicle whose door status is open based on the image features of the vehicles in the first image set; Distance unit: Used to determine a first key coordinate based on a preset image coordinate system of the acquired image; wherein the first key coordinate indicates the position of the car door when it is open; determine the orientation type of the key vehicle based on the correspondence between the image features of the vehicle and the vehicle orientation type; wherein the vehicle orientation type includes a first orientation type and / or a second orientation type, the vehicle orientation indicated by the first orientation type is different from the vehicle orientation indicated by the second orientation type; determine a first relative relationship between the car door when it is open and the key vehicle based on the first key coordinate; and determine the first relative relationship between the car door when it is open and the key vehicle based on the preset correspondence between the vehicle orientation type and the vehicle driving position. Based on the first and second relative relationships, a second relative relationship is determined between the driving position of the key vehicle and the key vehicle itself; based on the first and second relative relationships, a relative relationship is determined between the open door and the driving position of the key vehicle; in response to the opposite relative relationship between the open door and the driving position of the key vehicle, the open door is determined as a designated door, and a target distance is determined between the target object and the key vehicle; wherein, the designated door is used for the target object to enter and / or leave the key vehicle, and the target object is detected from the key vehicle in the associated area of ​​the acquired image; Anomaly Unit: Used to determine that the critical vehicle is an abnormal vehicle based on the target distance.

8. A readable storage medium, characterized in that, include, memory, The memory is used to store instructions that, when executed by a processor, cause a device including the readable storage medium to perform the method as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method as described in any one of claims 1-6.

Images