Method, device, electronic device and storage medium for detecting illegal parking of shared bicycles

Abstract

The present application relates to the field of intelligent transportation technology, and in particular to a shared bicycle illegal parking detection method, device, electronic device and storage medium to improve the detection accuracy of shared bicycle illegal parking. It includes: performing instance detection on each frame image in the video to be detected in turn, determining the target frame image containing non-motor vehicles, performing instance segmentation on the non-motor vehicle in the target frame image, and obtaining the body mask sub-image and the front mask sub-image of the non-motor vehicle; inputting the body mask sub-image and the front mask sub-image into the pre-trained feature extraction network to obtain the first feature; comparing the first feature with at least one second feature in the target feature set, respectively, to determine that the non-motor vehicle is a shared bicycle corresponding to the specific style represented by the second feature; by performing instance tracking on a continuous multi-frame image containing the target frame image, determining that the shared bicycle is in a non-use state, and performing illegal parking detection on the shared bicycle in any frame of the continuous multi-frame image.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a method, device, electronic device and storage medium for detecting illegal parking of shared bicycles. Background Art

[0002] In recent years, as shared bicycles have become popular across the country, the phenomenon of random parking has brought many negative impacts on the city's appearance, which is not conducive to the construction and development of urban civilization. The phenomenon of illegal parking of shared bicycles can be roughly divided into three situations: shared bicycles are not parked in designated areas, shared bicycles fall over, and shared bicycles are facing inconsistently. Therefore, there is an urgent need for a method that can identify illegal parking of shared bicycles, automatically identify illegal parking incidents, and notify city managers.

[0003] In the related art, when detecting illegal parking of shared bicycles, the brand of the shared bicycles is not determined. Since there are many brands of shared bicycles, the appearance of each brand of shared bicycles is different, and there are cases where shared bicycles and private non-motor vehicles are similar in appearance. Therefore, the detection method in the related art cannot distinguish between shared bicycles and private non-motor vehicles, and it is easy to mistakenly detect illegal parking of private non-motor vehicles of similar styles, resulting in low detection accuracy.

[0004] In summary, how to improve the detection accuracy of illegal parking of shared bicycles is an urgent issue to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and storage medium for detecting illegal parking of shared bicycles, so as to improve the detection accuracy of illegal parking of shared bicycles.

[0006] The embodiment of the present application provides a method for detecting illegal parking of a shared bicycle, the method comprising:

[0007] Perform instance detection on each frame image in the video to be detected in turn, determine a target frame image containing a non-motor vehicle, and perform instance segmentation on the non-motor vehicle in the target frame image to obtain a body mask sub-image and a front mask sub-image of the non-motor vehicle;

[0008] Inputting the vehicle body mask sub-image and the vehicle head mask sub-image into a pre-trained feature extraction network to obtain a first feature representing the style of the non-motor vehicle;

[0009] The first feature is compared with at least one second feature in the target feature set; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, the non-motor vehicle is determined to be a shared bicycle corresponding to a specific style represented by the second feature; each second feature is obtained by extracting features of a specific style of shared bicycles in advance through the feature extraction network; the specific style includes a specific color, a specific structure and a specific shape;

[0010] If it is determined that the shared bicycle is not in use by performing instance tracking on a plurality of consecutive image frames including the target image frame, illegal parking detection is performed on the shared bicycle in any one of the plurality of consecutive image frames.

[0011] The embodiment of the present application provides a shared bicycle illegal parking detection device, the device comprising:

[0012] An instance unit is used to perform instance detection on each frame image in the video to be detected in turn, determine a target frame image containing a non-motor vehicle, and perform instance segmentation on the non-motor vehicle in the target frame image to obtain a body mask sub-image and a front mask sub-image of the non-motor vehicle;

[0013] A feature extraction unit, configured to input the vehicle body mask sub-image and the vehicle head mask sub-image into a pre-trained feature extraction network to obtain a first feature representing the style of the non-motor vehicle;

[0014] A determination unit, configured to compare the first feature with at least one second feature in a target feature set; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, then determining that the non-motor vehicle is a shared bicycle corresponding to a specific style represented by the second feature; each of the second features is obtained by extracting features of a shared bicycle of a specific style in advance through the feature extraction network; the specific style includes a specific color, a specific structure, and a specific shape;

[0015] A detection unit is used to perform illegal parking detection on the shared bicycle in any frame of the continuous multiple frames if it is determined that the shared bicycle is in a non-use state by instance tracking of the continuous multiple frames of images including the target frame image; the illegal parking detection includes: at least one of parking area detection, vehicle head orientation detection and vehicle body posture detection.

[0016] Optionally, the detection unit is specifically used for:

[0017] If it is determined that the shared bicycle is in a stationary state by performing instance tracking on the continuous multiple frames of images, performing instance detection on the last frame of the continuous multiple frames of images;

[0018] If no human body is detected in the last frame of image, it is determined that the shared bicycle is in a non-use state;

[0019] Alternatively, if a human body is detected in the last frame of the image, and the overlap ratio between each human body and the shared bicycle is not greater than a preset overlap threshold, it is determined that the shared bicycle is in a non-use state.

[0020] Optionally, the detection unit is further used for:

[0021] If it is determined that the shared bicycle is in a non-stationary state by performing instance tracking on the continuous multiple frames of images, then no illegal parking detection is performed on the shared bicycle;

[0022] Alternatively, if a human body is detected in the last frame image, and the overlap ratio between at least one human body and the shared bicycle is greater than the overlap threshold, it is determined that the shared bicycle is in use, and the shared bicycle is not detected for illegal parking.

[0023] Optionally, the illegal parking detection includes: at least one of parking area detection and vehicle head direction detection; the detection unit is specifically used for:

[0024] Query the target object for the standard parking information pre-configured for the shared bicycle;

[0025] If the standard parking information is found, the illegal parking detection is performed on the shared bicycle based on the standard parking information;

[0026] If the standard parking information is not found, the standard parking information is determined by a pre-trained scene segmentation network, and the illegal parking detection is performed on the shared bicycle based on the standard parking information.

[0027] Optionally, the illegal parking detection includes parking area detection, the standard parking information includes a standard parking area, and the instance unit is specifically used to:

[0028] Perform instance detection on each frame image in the video to be detected in turn, and determine a target frame image containing a shared bicycle and a coordinate frame of the shared bicycle;

[0029] And, the detection unit is specifically used for:

[0030] The center coordinates of the coordinate frame are used as the position coordinates of the shared bicycle;

[0031] If the location coordinates are within the standard parking area, it is determined that the shared bicycle is in the standard parking area;

[0032] If the location coordinates are not within the standard parking area, it is determined that the shared bicycle is in an illegal parking area.

[0033] Optionally, the illegal parking detection includes parking area detection, the standard parking information includes a standard parking area corresponding to a specific style of the shared bicycle, and the detection unit is specifically used to:

[0034] Input any frame of the image into a pre-trained scene segmentation network, and obtain a pedestrian mask sub-image according to the corresponding segmentation result;

[0035] The standard parking area is generated according to the sidewalk mask subgraph, and the standard parking area represents a closed area formed by surrounding the sidewalk.

[0036] Optionally, before determining that the shared bicycle is in the standard parking area if the location coordinates are in the standard parking area, the detection unit is further used to:

[0037] If the position coordinates are not in the motor vehicle lane area, determining whether the position coordinates are in the non-motor vehicle lane area;

[0038] If the location coordinates are not in the non-motorized vehicle lane area, determining whether the location coordinates are in the sidewalk area;

[0039] If the position coordinates are within the sidewalk area, it is determined whether the position coordinates are within the standard parking area of ​​the sidewalk area.

[0040] Optionally, the illegal parking detection includes vehicle head direction detection, and the standard parking information includes a second direction vector, and the detection unit is specifically configured to:

[0041] A first direction vector is constructed by taking the center point of the vehicle body mask sub-image as the starting point of the vector and taking the center point of the vehicle head mask sub-image as the end point of the vector;

[0042] Determine the angle between the first direction vector and the second direction vector; the second direction vector represents the standard vehicle head direction;

[0043] If the angle is less than a preset angle threshold, the front direction of the shared bicycle is determined to be in a standard direction;

[0044] If the angle is not less than the preset angle threshold, it is determined that the front direction of the shared bicycle is in violation of the regulations.

[0045] Optionally, the illegal parking detection includes vehicle head direction detection, and the standard parking information includes a second direction vector, and the detection unit is specifically configured to:

[0046] Inputting any frame of the image into a pre-trained scene segmentation network, and obtaining a pedestrian mask sub-image and a road mask sub-image according to the corresponding segmentation results;

[0047] The longest boundary line between the pedestrian mask sub-image and the road mask sub-image is used as the target boundary line;

[0048] A direction vector that is perpendicular to the target boundary line and points to the lane mask sub-image is used as the second direction vector.

[0049] Optionally, the illegal parking detection includes vehicle body posture detection, and the detection unit is specifically used for:

[0050] Inputting the first feature into a pre-trained attribute classification network to obtain a classification value;

[0051] If the classification value is greater than a preset classification threshold, it is determined that the shared bicycle is in a lying state;

[0052] If the classification value is not greater than the preset classification threshold, it is determined that the shared bicycle is in an upright state.

[0053] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned shared bicycle illegal parking detection methods.

[0054] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned shared bicycle illegal parking detection methods.

[0055] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned shared bicycle illegal parking detection methods.

[0056] The beneficial effects of this application are as follows:

[0057] The embodiments of the present application provide a method, device, electronic device and storage medium for detecting illegal parking of shared bicycles. In the embodiments of the present application, firstly, each frame image in the video to be detected is subjected to instance detection in turn, a target frame image containing a non-motor vehicle is determined, and the non-motor vehicle in the target frame image is subjected to instance segmentation to obtain a body mask sub-image and a front mask sub-image of the non-motor vehicle; since the mask image does not contain an image of the background in which the non-motor vehicle is located, as well as images of other non-motor vehicles, the body mask sub-image and the front mask sub-image are input into a pre-trained feature extraction network to obtain a first feature representing the style of the non-motor vehicle, and the first feature only contains the features of the non-motor vehicle, avoiding interference from the background and other non-motor vehicles;

[0058] Then, the first feature is compared with at least one second feature in the target feature set; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, the non-motor vehicle is determined to be a shared bicycle corresponding to the style represented by the second feature, and the non-motor vehicle is determined to be a shared bicycle that needs to be detected for illegal parking, not a private non-motor vehicle. Finally, by tracking the instance to determine that the shared bicycle is not in use, it is avoided that the shared bicycle is detected for illegal parking after the shared bicycle passes through the area during driving.

[0059] Therefore, the embodiment of the present application eliminates various interference situations through rigorous logic, and after accurately identifying the shared bicycles to be detected, provides a variety of illegal parking detections for shared bicycles, basically covering all situations of illegal parking of shared bicycles. It has strong practicality and meets the needs of different scenarios.

[0060] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0062] Figure 1 A schematic diagram of an application scenario of a shared bicycle illegal parking detection method provided in an embodiment of the present application;

[0063] Figure 2 A flowchart of a method for detecting illegal parking of a shared bicycle provided in an embodiment of the present application;

[0064] Figure 3A schematic diagram of instance detection and instance segmentation provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of a standardized parking area provided in an embodiment of the present application;

[0066] Figure 5 A flowchart of a parking area detection provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of a second direction vector provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of another second direction vector provided in an embodiment of the present application;

[0069] Figure 8 A flow chart of vehicle head orientation detection provided in an embodiment of the present application;

[0070] Fig. 9 A flow chart of vehicle body posture detection provided in an embodiment of the present application;

[0071] Fig.10 A flowchart of another method for detecting illegal parking of shared bicycles provided in an embodiment of the present application;

[0072] Fig.11 This is a structural schematic diagram of a shared bicycle illegal parking detection device in an embodiment of the present application;

[0073] Fig.12 A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0074] Fig.13 A schematic diagram of the hardware composition structure of a computing device using an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0076] The following is an introduction to some concepts involved in the embodiments of the present application.

[0077] Sidewalk: refers to the part of the road separated by curbs, guardrails and other similar facilities for pedestrians to pass, usually located on both sides of the carriageway, different from zebra crossing.

[0078] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0079] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes a video acquisition device 110, a server 120 and a terminal device 130.

[0080] The method for detecting illegal parking of shared bicycles provided in the embodiment of the present application can be jointly executed by the video acquisition device 110, the server 120 and the terminal device 130. The video acquisition device 110 collects images of areas where shared bicycles may be parked, such as the sides of the road, and sends the collected video to be detected to the server 120. The target object sends information such as pictures of the shared bicycles to be detected to the server 120 through the terminal device 130. The server 120 performs instance detection on each frame image in the video to be detected in turn, determines the target frame image containing the non-motor vehicle, and performs instance segmentation on the non-motor vehicle in the target frame image to obtain the body mask sub-image and the front mask sub-image of the non-motor vehicle; then, the server 120 inputs the body mask sub-image and the front mask sub-image into the pre-trained feature extraction network , obtain a first feature that characterizes the style of the non-motor vehicle; and input the received shared bicycle image sent by the terminal device 130 into the feature extraction network to obtain a second feature, and the server 120 compares the first feature with at least one second feature in the target feature set respectively; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, the non-motor vehicle is determined to be a shared bicycle corresponding to the specific style characterized by the second feature; if the server 120 determines that the shared bicycle is in a non-use state by instance tracking of multiple consecutive frames of images containing the target frame image, it will perform illegal parking detection on the shared bicycle in any frame of the multiple consecutive frames, and send the detection result to the terminal device 130, so that the terminal device 130 presents the detection result to the target object.

[0081] In an optional implementation, the video acquisition device 110, the server 120 and the terminal device 130 may communicate with each other via a communication network.

[0082] In an optional implementation, the communication network is a wired network or a wireless network.

[0083] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0084] The following describes the shared bicycle illegal parking detection method provided by the exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate the understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0085] See also Figure 2 As shown, it is an implementation flow chart of a shared bicycle illegal parking detection method provided in an embodiment of the present application. The specific implementation process of the method includes S201~S204:

[0086] S201: Perform instance detection on each frame image in the video to be detected in turn, determine the target frame image containing the non-motor vehicle, and perform instance segmentation on the non-motor vehicle in the target frame image to obtain the body mask sub-image and the front mask sub-image of the non-motor vehicle.

[0087] The video to be detected is obtained by collecting and decoding a video acquisition device installed in an area where shared bicycles may be parked. The video acquisition device may be a smart camera, etc. The area where shared bicycles may be parked may be a sidewalk in a city, etc. Non-motor vehicles include bicycles and electric vehicles.

[0088] For example, smart cameras can be installed near city sidewalks. The smart cameras can capture real-time images on both sides of the road, decode them using video encoding and decoding technology, and then send the decoded data of the real-time images to the server.

[0089] In an embodiment of the present application, instance detection and instance segmentation are implemented by a target detection (also known as instance detection) and instance segmentation network, which is built based on the You Only Look Once version 8 (YOLOv8) framework. YOLOv8 can implement instance detection and instance segmentation tasks, and these two tasks share feature extraction network weights.

[0090] Among them, instance detection can detect two types of instances in the image: human body and non-motor vehicle, and can also detect the coordinate frame of each type of instance.

[0091] In the embodiment of the present application, when acquiring the target frame image, an optional implementation method is as follows:

[0092] Perform instance detection on each frame image in the video to be detected in turn, and determine the target frame image containing the shared bicycle and the coordinate frame of the shared bicycle;

[0093] Furthermore, considering that the feature extraction of the non-motor vehicle part in the target frame image is required in the subsequent steps, if the detection frame is used for feature extraction, the feature extraction result will be interfered by the background part in the detection frame. In addition, non-motor vehicles are often placed side by side, resulting in the inclusion of other closer non-motor vehicles in the detection frame. Therefore, the embodiment of the present application will perform instance segmentation on the non-motor vehicle in the image, and divide the entity of the non-motor vehicle into two parts: the front and the body. Both parts are mask sub-images (masks) and do not include instances other than the non-motor vehicle.

[0094] like Figure 3 As shown, it is a schematic diagram of instance detection and instance segmentation provided in an embodiment of the present application. Figure 3 The upper middle figure is a target frame image containing a non-motor vehicle obtained after instance detection, where S301 is the coordinate frame of the non-motor vehicle. Figure 3 The lower middle figure is an image after the non-motor vehicle is segmented. The non-motor vehicle is segmented into two parts: a vehicle body mask sub-image shown in S302 and a vehicle head mask sub-image shown in S303.

[0095] Therefore, after receiving the video to be detected, the server transmits the image data of each frame in the video to be detected to the target detection and instance segmentation network, and can determine the target frame image containing the non-motor vehicle, as well as the body mask sub-image and the front mask sub-image of the non-motor vehicle.

[0096] In the above implementation, the embodiment of the present application obtains a body mask sub-image and a front mask sub-image containing only non-motor vehicles through instance detection and instance segmentation, thereby avoiding interference from the background or other non-motor vehicles and improving the accuracy of subsequent feature extraction and other steps.

[0097] After obtaining the mask image of the non-motor vehicle in the above manner, the feature extraction of the non-motor vehicle can be performed through the following steps:

[0098] S202: Input the vehicle body mask sub-image and the vehicle head mask sub-image into a pre-trained feature extraction network to obtain a first feature representing the style of the non-motor vehicle.

[0099] Among them, the feature extraction network can be a lightweight convolutional neural network, such as the mobilenet-v3 lightweight network.

[0100] Specifically, the vehicle body mask sub-image and the vehicle head mask sub-image obtained by instance segmentation in step S21 are merged, input into a feature extraction network, and a feature extraction operation is performed to obtain a first feature, which is a floating point value of 512 dimensions.

[0101] After obtaining the features of the non-motor vehicle through the above steps, the following steps can be used to determine whether the non-motor vehicle is a shared bicycle:

[0102] S203: Compare the first feature with at least one second feature in the target feature set respectively; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, determine that the non-motor vehicle is a shared bicycle corresponding to the specific style represented by the second feature.

[0103] Among them, the specific styles include specific colors, specific structures and specific shapes.

[0104] Specifically, the styles of shared bicycles in different regions and brands are different. For example, shared bicycles of different brands have their own specific colors, specific structures and shapes, etc., while the styles of private non-motor vehicles do not have such characteristics. They often have various styles and are quite different from those of shared bicycles.

[0105] Therefore, before comparing non-motor vehicles, it is necessary to receive a base image of at least one shared bicycle uploaded by the target object, and perform instance segmentation on each shared bicycle base image through the above-mentioned target detection (that is, instance detection) and instance segmentation network. The feature extraction network then merges the segmented front mask subimage and the body mask subimage and performs feature extraction to obtain the corresponding second feature, thereby obtaining a target feature set containing at least one second feature.

[0106] Then, the first feature is compared with each second feature in the target feature set. If the similarity between the first feature and a second feature reaches a similarity threshold, the non-motor vehicle can be considered to be a shared bicycle corresponding to the second feature.

[0107] Among them, the similarity threshold ( sim-th ) is an empirical value and can be 0.8, 0.9, etc.

[0108] In the embodiment of the present application, when comparing the first feature and the second feature, an optional implementation is shown in the following formula 1:

[0109] (Formula 1)

[0110] in, feature1 The first feature is feature2 For the second feature, i For the dimension, sim For similarity.

[0111] Will sim and sim-th For comparison, if sim > sim-th , then the non-motor vehicle is considered to be feature2For the corresponding shared bicycle, the subsequent illegal parking detection can be carried out on the non-motor vehicle. If the similarity between the non-motor vehicle and any second feature in the target feature set reaches the threshold, it is considered that the non-motor vehicle may be a private non-motor vehicle, or other types of shared bicycles that the target object is not concerned about. In this case, the non-motor vehicle is filtered and no subsequent illegal parking detection is carried out.

[0112] In the above implementation, in view of the different styles of shared bicycle brands in different regions across the country, by uploading a shared bicycle base map, feature extraction and feature comparison, the target object can arbitrarily set the shared bicycles that need to be detected, filter out private non-motor vehicles and other styles of shared bicycles that the target object is not concerned about, avoid misdetection of them, and accurately identify the specific style of shared bicycles that the target object is concerned about.

[0113] After determining that the non-motor vehicle is a shared bicycle that needs to be detected in the above manner, the embodiment of the present application performs illegal parking detection on the shared bicycle in the following manner:

[0114] S204: If it is determined that the shared bicycle is not in use by performing instance tracking on a plurality of consecutive image frames including the target image frame, illegal parking detection is performed on the shared bicycle in any one of the plurality of consecutive image frames.

[0115] Considering that the shared bicycle in the target frame image may be in use and is only captured by the video acquisition device, the embodiment of the present application determines whether the shared bicycle is in use in the following manner before performing illegal parking detection on the shared bicycle:

[0116] Step 1: If it is determined that the shared bicycle is in a stationary state by performing instance tracking on multiple consecutive frames of images, instance detection is performed on the last frame of the multiple consecutive frames of images; if it is determined that the shared bicycle is in a non-stationary state by performing instance tracking on multiple consecutive frames of images, illegal parking detection is not performed on the shared bicycle.

[0117] Specifically, the displacement of the non-motor vehicle is obtained by tracking multiple consecutive frames of images; if the displacement is less than a preset displacement threshold, it is determined that the non-motor vehicle is in a stationary state.

[0118] In an optional implementation manner, the displacement formula is shown in the following formula 2:

[0119] (Formula 2)

[0120] in, S is the displacement, N is the number of frames. The multi-frame tracking in the embodiment of the present application is generally a medium-length sequence tracking. NThe value can be 10~50 frames, or it can be long sequence tracking. N The value is 50 frames or more. x , y ) is the center point of the coordinate frame.

[0121] When S<displacement threshold ( S-th ), it can be considered that the shared bicycle is in a stationary state, so the next step is to judge the shared bicycle; when S ≥ S-th , it can be considered that the shared bicycle is in motion, that is, the shared bicycle is being used by someone, so the shared bicycle will be filtered and no illegal parking detection will be performed.

[0122] In the above implementation, the displacement of shared bicycles is compared through the displacement threshold, and shared bicycles in motion are filtered out, thereby reducing unnecessary illegal parking detections, improving the efficiency of illegal parking detections, and avoiding waste of resources.

[0123] Step 2: If no human body is detected in the last frame of the image, it is determined that the shared bicycle is not in use; or, if a human body is detected in the last frame of the image, and the overlap ratio between each human body and the shared bicycle is not greater than the preset overlap threshold, it is determined that the shared bicycle is not in use. If a human body is detected in the last frame of the image, and there is at least one human body and the shared bicycle The overlap ratio is greater than the overlap threshold, it is determined that the shared bicycle is in use, and the shared bicycle is not detected for illegal parking.

[0124] The overlap threshold is 0. In the embodiment of the present application, the overlap ratio between the human body and the shared bicycle is determined by calculating the intersection-and-union ratio.

[0125] Specifically, when the last frame of multiple consecutive frames is detected by the target detection and instance segmentation network, if a human body is detected in the last frame, the intersection-and-union (iou) of each human body and the shared bicycle in a static state is calculated. Since the value of iou is [0, 1], when the iou of each human body and the shared bicycle is 0, it can be considered that the shared bicycle is not used by any person and is in a non-use state. If iou>0, it is considered that the human body and the shared bicycle have overlapping parts, and the shared bicycle may be ridden by the object, but it is only captured by the video capture device when it is parked on the roadside. Therefore, no illegal parking detection is performed.

[0126] In the above implementation, the shared bicycles in use are filtered out through the intersection-and-union ratio calculation, which reduces unnecessary illegal parking detection, improves the efficiency of illegal parking detection, and avoids waste of resources.

[0127] After the shared bicycles that are not in use are identified in the above manner, illegal parking detection can be performed on the shared bicycles in the following manner:

[0128] Specifically, illegal parking detection includes at least one of parking area detection, vehicle head orientation detection and vehicle body posture detection.

[0129] In the embodiment of the present application, when the illegal parking detection includes at least one of the parking area detection and the vehicle head direction detection, an optional implementation method for performing illegal parking detection on the shared bicycle is as follows:

[0130] The target object is queried for the pre-configured standardized parking information for shared bicycles; if the standardized parking information is found, illegal parking detection is performed on the shared bicycles based on the standardized parking information; if the standardized parking information is not found, the standardized parking information is determined through the pre-trained scene segmentation network, and illegal parking detection is performed on the shared bicycles based on the standardized parking information.

[0131] Specifically, for parking area detection and vehicle head direction detection, the embodiment of the present application provides two detection modes. Mode 1 is that the target object is pre-configured with standard parking information, and the shared bicycle is directly detected for illegal parking according to the standard parking information; Mode 2 is that the target object is not pre-configured with standard parking information, and the embodiment of the present application can automatically generate standard parking information, and then detect the shared bicycle for illegal parking according to the standard parking information. Among them, the priority of mode 1 is higher than the priority of mode 2.

[0132] Since different brands have different parking area plans for their shared bicycles in different regions, when illegal parking detection is a parking area detection, the standard parking information includes the standard parking area corresponding to the specific style of the shared bicycle. The following is a detailed introduction to the parking area detection in illegal parking detection:

[0133] The center coordinates of the coordinate frame are used as the location coordinates of the shared bicycle; if the location coordinates are within the standard parking area, it is determined that the shared bicycle is in the standard parking area; if the location coordinates are not within the standard parking area, it is determined that the shared bicycle is in the illegal parking area.

[0134] Specifically, determine the center coordinates of the coordinate frame ( x , y ) is in the specified parking area. If so, the parking area of ​​the shared bicycle can be considered compliant. Otherwise, the parking area of ​​the shared bicycle can be considered illegal, and the violation will be reported.

[0135] In addition, when determining whether the shared bicycle is in a standard parking area, the possible current parking locations of the shared bicycle may be narrowed down first, and then it is determined whether it is in the standard parking area.

[0136] In the embodiment of the present application, when the parking area of ​​the shared bicycle is detected, an optional implementation method is as follows:

[0137] If the location coordinates are not in the motor vehicle lane area, determine whether the location coordinates are in the non-motor vehicle lane area; if the location coordinates are not in the non-motor vehicle lane area, determine whether the location coordinates are in the sidewalk area; if the location coordinates are in the sidewalk area, determine whether the location coordinates are in the standard parking area of ​​the sidewalk area.

[0138] Specifically, the location coordinates of the shared bicycle are judged in turn with the motor vehicle lane area, non-motor vehicle lane area, and sidewalk area. If the location coordinates of the shared bicycle are in the motor vehicle lane area, it can be determined that the parking area of ​​the shared bicycle is incorrect. If the location coordinates of the shared bicycle are not in the motor vehicle lane but in the non-motor vehicle lane, it can also be determined that the parking area of ​​the shared bicycle is incorrect. When it is determined that the shared bicycle is parked in the sidewalk area, it can be further determined whether the shared bicycle is in the standard parking area of ​​the sidewalk area.

[0139] In addition, it should be noted that, based on actual needs, if the standard parking area is not on the sidewalk, you can also add or modify the judgment steps. For example, if the standard parking area is a parking lot, you can judge the motor vehicle lane area, non-motor vehicle lane area and parking lot area in turn, which will not be elaborated here.

[0140] If the target object is not configured with a standard parking area, an optional implementation method of the embodiment of the present application when automatically generating a standard parking area is as follows:

[0141] Input any frame of image into the pre-trained scene segmentation network, and obtain the sidewalk mask sub-map according to the corresponding segmentation result; generate a standard parking area according to the sidewalk mask sub-map, and the standard parking area represents the closed area formed by surrounding the sidewalk.

[0142] Among them, the scene segmentation network can use deeplab-v3. After the image is input into the scene segmentation network, the types that can be segmented include roadways, sidewalks, buildings, skies, vegetation and other scene background types.

[0143] Specifically, for the area captured by the video acquisition device, the scene background in the area will not change, so any frame image in the multiple consecutive frames can be input into the scene segmentation network. For example, the current frame image can be input into the scene segmentation network, and the sidewalk mask sub-image of the segmented sidewalk is selected as the target area, and the mask is used to generate a closed area as the standard parking area.

[0144] like Figure 4 As shown, it is a schematic diagram of a standardized parking area provided in an embodiment of the present application. Figure 4 The upper middle picture is the current frame image. Figure 4 The white area in the lower middle picture is the sidewalk area, the dark area in the middle of the white area is the driveway area, and the closed polygon area generated based on the white area is the standard parking area.

[0145] like Figure 5 As shown, it is a flowchart of a parking area detection provided by an embodiment of the present application, specifically including S501~S507:

[0146] S501: Querying the pre-configured standard parking area of ​​the target object;

[0147] Specifically, if the query is found, jump to S504; if the query is not found, jump to S502.

[0148] S502: Obtain the video to be detected;

[0149] Specifically, any frame of the video to be detected is input into the scene segmentation network.

[0150] S503: sidewalk mask sub-graph;

[0151] S504: Standardize parking areas;

[0152] S505: Determine whether the shared bicycle is in a standard parking area;

[0153] Specifically, if the shared bicycle is in the standard parking area, jump to S506; otherwise, jump to S507.

[0154] S506: The shared bicycle is in the specified parking area.

[0155] S507: The shared bicycle is in an illegal parking area.

[0156] In the above implementation, the parking area of ​​the shared bicycle is detected by pre-configuring a standardized parking area for the target object or automatically generating a commission area. Implementing this detection in two modes greatly improves the usability of the solution, reduces the manpower cost of deployment, and makes the detection smarter and more convenient to use.

[0157] When the illegal parking detection is a vehicle head orientation detection, the standard parking information includes a second direction vector representing the standard vehicle head orientation. The following is a detailed introduction to the parking area detection in illegal parking detection:

[0158] A first direction vector is constructed with the center point of the vehicle body mask sub-image as the starting point of the vector and the center point of the vehicle head mask sub-image as the end point of the vector; the angle between the first direction vector and the second direction vector is determined; the second direction vector represents the standard vehicle head orientation; if the angle is less than the preset angle threshold, the vehicle head orientation of the shared bicycle is determined to be standard; if the angle is not less than the preset angle threshold, the vehicle head orientation of the shared bicycle is determined to be illegal.

[0159] The preset angle threshold is an empirical value, and a smaller angle value such as 10 degrees or 20 degrees may be used.

[0160] Specifically, in any frame image, such as the current frame image, a standard vehicle head direction is drawn, that is, a second direction vector from the rear of the vehicle to the front of the vehicle a . Generate the center point c1 based on the mask sub-graph of the shared bicycle’s front face ( x 1 , y 1 ), the body mask sub-image generates the center point c2 ( x 2 , y 2 ), get the first direction vector b =( x 1 - x 2 , y 1 - y 2 )

[0161] In the embodiment of the present application, when determining the angle between the first direction vector and the second direction vector, an optional implementation is shown in the following formula 3:

[0162] (Formula 3)

[0163] If the angle α <Preset Angle Threshold α - th , it can be considered that the front of the shared bicycle is facing the standard. If the angle α ≥Preset angle threshold α - th , it can be considered that the front direction of the shared bicycle is in violation of the regulations and the violation should be reported.

[0164] If the target object is pre-configured with a standard vehicle head orientation, the second direction vector is directly drawn in the captured scene.

[0165] like Figure 6 As shown, it is a schematic diagram of a second direction vector provided in an embodiment of the present application. Figure 6The white arrow in the middle is the standard vehicle head orientation represented by the second direction vector, pointing from the sidewalk to the roadway.

[0166] If the target object is not pre-configured with a standard vehicle head orientation, that is, the second direction vector is not pre-configured, the embodiment of the present application automatically generates the second direction vector in the following manner:

[0167] Input any frame of image into the pre-trained scene segmentation network, and obtain the sidewalk mask submap and the road mask submap according to the corresponding segmentation results; take the longest boundary line between the sidewalk mask submap and the road mask submap as the target boundary line; take the direction vector perpendicular to the target boundary line and pointing to the road mask submap as the second direction vector.

[0168] Specifically, any frame of image is input into the scene segmentation network to obtain the sidewalk mask subgraph and the roadway mask subgraph, and then a closed regular area S1 is generated according to the roadway mask subgraph, and a closed regular area S2 is generated according to the sidewalk mask subgraph. The line segment of the intersection between closed areas S1 and S2 is the boundary line. Considering that there may be multiple boundary lines, the longest one is taken as the target boundary line, and the vector perpendicular to the target boundary line and pointing from S2 to S1 is taken as the second direction vector.

[0169] like Figure 7 As shown, it is a schematic diagram of another second direction vector provided in an embodiment of the present application. Figure 7 The dark area is S1, the white area is S2, the thick straight line is the target boundary line, and the direction of the arrow is the direction of the automatically generated second direction vector.

[0170] like Figure 8 As shown, it is a flow chart of a vehicle head direction detection provided by an embodiment of the present application, specifically including S801~S811:

[0171] S801: Obtain the video to be detected;

[0172] S802: vehicle body mask sub-image, vehicle head mask sub-image;

[0173] S803: first direction vector;

[0174] Specifically, a first direction vector is constructed with the center point of the vehicle body mask sub-image as the starting point of the vector and the center point of the vehicle head mask sub-image as the end point of the vector.

[0175] S804: Querying a second direction vector pre-configured for the target object;

[0176] Specifically, if the query is found, jump to S808; if the query is not found, jump to S805.

[0177] S805: Obtain the video to be detected;

[0178] Specifically, any frame of the video to be detected is input into the scene segmentation network.

[0179] S806: pedestrian mask sub-map, road mask sub-map;

[0180] S807: perpendicular to the boundary line, pointing to the road mask sub-graph;

[0181] S808: second direction vector;

[0182] S809: Determine whether the angle is less than an angle threshold;

[0183] Specifically, if the included angle is less than the angle threshold, jump to S810; otherwise, jump to S811.

[0184] S810: Specifications for the front orientation of shared bicycles;

[0185] S811: The front of the shared bicycle is facing in a illegal direction.

[0186] In the above implementation, the front orientation of the shared bicycle is detected by pre-configuring a standard front orientation for the target object or automatically generating a standard front orientation. Implementing this detection in two modes greatly improves the usability of the solution, reduces the manpower cost of deployment, and makes the detection smarter and more convenient to use.

[0187] The following is a detailed introduction to the vehicle body posture detection in illegal parking detection:

[0188] The first feature is input into the pre-trained attribute classification network to obtain a classification value; if the classification value is greater than a preset classification threshold, it is determined that the shared bicycle is in a lying state; if the classification value is not greater than the preset classification threshold, it is determined that the shared bicycle is in an upright state.

[0189] Among them, the preset classification threshold can be 0.5.

[0190] Specifically, the first feature with a dimension of (1, 512) is input into the attribute classification network. The network consists of only two layers, namely a fully connected layer and a softmax layer. During the classification process, the first feature passes through the first fully connected layer with a dimension from (1, 512) to (1, 64), passes through the second fully connected layer with a dimension from (1, 64) to (1, 2), and finally passes through the softmax layer to obtain the probability of lying down conf. If conf>0.5, the classification result is judged to be a lying down state, and the shared bicycle judged to be in a lying down state is reported; if conf≤0.5, the classification result is judged to be an upright state.

[0191] like Fig. 9As shown, it is a flow chart of a vehicle body posture detection provided by an embodiment of the present application, specifically including S901~S904:

[0192] S901: first feature;

[0193] S902: attribute classification network;

[0194] S903: Verify classification category;

[0195] S904: Determine the lodging state;

[0196] Specifically, if conf>0.5, the classification result is determined to be in a lying state, and the shared bicycle determined to be in a lying state is reported; if conf≤0.5, the classification result is determined to be in an upright state.

[0197] In the above implementation, the feature reuse method not only reduces the amount of calculation, but also uses the features of the mask image to focus more on the shared bicycle itself (removing the influence of background and overlapping vehicles).

[0198] After the above steps, we can obtain the events of shared bicycles not being parked in the designated area, shared bicycles with irregular front directions, and shared bicycles falling over. The detection results are integrated and reported to the platform business layer, and law enforcement personnel are notified to handle them.

[0199] like Fig.10 As shown, it is an implementation flow chart of a shared bicycle illegal parking detection method provided in an embodiment of the present application, which specifically includes S1001~S1014:

[0200] S1001: Shared bicycle base map;

[0201] Specifically, the shared bicycle base map provided by the target object is input into the target detection and instance segmentation network.

[0202] S1002: vehicle body mask sub-image and vehicle head mask sub-image;

[0203] Specifically, the vehicle body mask sub-image and the vehicle front mask sub-image are merged and input into the feature extraction network.

[0204] S1003: target feature set;

[0205] S1004: Obtain the video to be detected;

[0206] S1005: vehicle body mask sub-image and vehicle head mask sub-image;

[0207] Specifically, the vehicle body mask sub-image and the vehicle front mask sub-image are merged and input into the feature extraction network.

[0208] S1006: First feature;

[0209] S1007: Determine whether the feature similarity reaches a similarity threshold;

[0210] Specifically, if the feature similarity does not reach the similarity threshold, jump to S1008; otherwise, jump to S1009.

[0211] S1008: Filter this non-motor vehicle;

[0212] S1009: Determine whether the shared bicycle is in a stationary state;

[0213] Specifically, if the shared bicycle is not in a stationary state, jump to S1010; otherwise, jump to S1011.

[0214] S1010: Filter this non-motor vehicle;

[0215] S1011: Determine whether the shared bicycle is in a non-use state;

[0216] Specifically, if the shared bicycle is in use, jump to S1012; otherwise, jump to S1013.

[0217] S1012: Filter this non-motor vehicle;

[0218] S1013: parking area detection, vehicle head orientation detection, and vehicle body posture detection;

[0219] S1014: Comprehensive detection results are fed back to the target object.

[0220] In the above implementation, the embodiment of the present application supports three illegal parking detection functions, namely parking area detection, vehicle head direction detection and vehicle body posture detection. It basically covers all situations of illegal parking of shared bicycles, has strong practicality, and meets the needs of different scenarios. In addition, the deep learning network in the embodiment of the present application has a lightweight network structure, shared weight parameters, and feature reuse, which reduces the amount of network calculation, requires low computing power, and reduces the hardware cost of deployment. In addition, the event logic judgment of the embodiment of the present application has progressive steps, strict logic, and strong practicality. Through multi-frame static judgment, human matching filtering conditions and other logic, it can more accurately capture illegal parking events.

[0221] Based on the same inventive concept, the present application also provides a shared bicycle illegal parking detection device, such as Fig.11 As shown, the shared bicycle illegal parking detection device 1100 includes:

[0222] The instance unit 1101 is used to perform instance detection on each frame image in the video to be detected in turn, determine the target frame image containing the non-motor vehicle, and perform instance segmentation on the non-motor vehicle in the target frame image to obtain a body mask sub-image and a front mask sub-image of the non-motor vehicle;

[0223] A feature extraction unit 1102 is used to input the vehicle body mask sub-image and the vehicle head mask sub-image into a pre-trained feature extraction network to obtain a first feature representing the style of the non-motor vehicle;

[0224] The determination unit 1103 is used to compare the first feature with at least one second feature in the target feature set respectively; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, the non-motor vehicle is determined to be a shared bicycle corresponding to a specific style represented by the second feature; each second feature is obtained by extracting features of a shared bicycle of a specific style in advance through a feature extraction network; the specific style includes a specific color, a specific structure and a specific shape;

[0225] The detection unit 1104 is used to perform illegal parking detection on the shared bicycle in any frame of the continuous multiple frames if it is determined that the shared bicycle is not in use by instance tracking the continuous multiple frames of images including the target frame image.

[0226] Optionally, the detection unit 1104 is specifically used for:

[0227] If it is determined that the shared bicycle is stationary by performing instance tracking on multiple consecutive image frames, instance detection is performed on the last frame of the multiple consecutive image frames;

[0228] If no human body is detected in the last frame of the image, it is determined that the shared bicycle is not in use;

[0229] Alternatively, if a human body is detected in the last frame of the image, and the overlap ratio between each human body and the shared bicycle is not greater than a preset overlap threshold, it is determined that the shared bicycle is in a non-use state.

[0230] Optionally, the detection unit 1104 is further configured to:

[0231] If the shared bicycle is determined to be in a non-stationary state by tracking the instance of multiple consecutive frames of images, the shared bicycle will not be detected for illegal parking;

[0232] Alternatively, if a human body is detected in the last frame image, and the overlap ratio between at least one human body and the shared bicycle is greater than the overlap threshold, it is determined that the shared bicycle is in use, and the shared bicycle is not detected for illegal parking.

[0233] Optionally, the illegal parking detection includes: at least one of parking area detection and vehicle head direction detection; the detection unit 1104 is specifically used for:

[0234] Query the target object's pre-configured standard parking information for shared bicycles;

[0235] If the standardized parking information is found, the shared bicycles will be checked for illegal parking based on the standardized parking information;

[0236] If no standardized parking information is found, the standardized parking information is determined through the pre-trained scene segmentation network, and illegal parking detection of shared bicycles is performed based on the standardized parking information.

[0237] Optionally, the illegal parking detection includes parking area detection, and the standard parking information includes a standard parking area corresponding to a specific style of the shared bicycle. The instance unit 1101 is specifically used for:

[0238] Perform instance detection on each frame image in the video to be detected in turn, and determine the target frame image containing the shared bicycle and the coordinate frame of the shared bicycle;

[0239] And, the detection unit 1104 is specifically used for:

[0240] The center coordinates of the coordinate frame are used as the location coordinates of the shared bicycle;

[0241] If the location coordinates are within the standard parking area, it is determined that the shared bicycle is in the standard parking area;

[0242] If the location coordinates are not within the standard parking area, it is determined that the shared bicycle is in an illegal parking area.

[0243] Optionally, if the location coordinates are within the standard parking area, before determining that the shared bicycle is within the standard parking area, the detection unit 1104 is further configured to:

[0244] If the location coordinates are not in the motor vehicle lane area, determine whether the location coordinates are in the non-motor vehicle lane area;

[0245] If the location coordinates are not in the non-motorized vehicle lane area, determine whether the location coordinates are in the sidewalk area;

[0246] If the location coordinates are within the sidewalk area, it is determined whether the location coordinates are within the standard parking area of ​​the sidewalk area.

[0247] Optionally, the illegal parking detection includes parking area detection, the standard parking information includes the standard parking area, and the detection unit 1104 is specifically used for:

[0248] Input any frame of image into the pre-trained scene segmentation network, and obtain the sidewalk mask sub-image according to the corresponding segmentation result;

[0249] A standard parking area is generated according to the sidewalk mask sub-graph, and the standard parking area represents a closed area formed by surrounding the sidewalk.

[0250] Optionally, the illegal parking detection includes vehicle head direction detection, and the standard parking information includes a second direction vector, and the detection unit 1104 is specifically used for:

[0251] The first direction vector is constructed by taking the center point of the vehicle body mask sub-image as the starting point of the vector and the center point of the vehicle head mask sub-image as the end point of the vector;

[0252] Determine the angle between the first direction vector and the second direction vector; the second direction vector represents the standard vehicle head direction;

[0253] If the angle is less than the preset angle threshold, the shared bicycle's front direction is determined to be standard;

[0254] If the angle is not less than the preset angle threshold, it is determined that the front direction of the shared bicycle is in violation of the regulations.

[0255] Optionally, the illegal parking detection includes vehicle head direction detection, and the standard parking information includes a second direction vector, and the detection unit 1104 is specifically used for:

[0256] Input any frame of image into the pre-trained scene segmentation network, and obtain the sidewalk mask sub-image and the roadway mask sub-image according to the corresponding segmentation results;

[0257] The longest boundary line between the pedestrian mask sub-image and the road mask sub-image is used as the target boundary line;

[0258] A direction vector that is perpendicular to the target boundary line and points to the road mask sub-image is used as the second direction vector.

[0259] Optionally, the illegal parking detection includes vehicle body posture detection, and the detection unit 1104 is specifically used for:

[0260] Input the first feature into the pre-trained attribute classification network to obtain the classification value;

[0261] If the classification value is greater than the preset classification threshold, the shared bicycle is determined to be in a lying state;

[0262] If the classification value is not greater than the preset classification threshold, it is determined that the shared bicycle is in an upright state.

[0263] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application. In one embodiment, the electronic device may be Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.12 As shown, it includes a memory 1201 , a communication module 1203 and one or more processors 1202 .

[0264] The memory 1201 is used to store computer programs executed by the processor 1202. The memory 1201 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0265] The memory 1201 may be a volatile memory, such as a random-access memory (RAM); the memory 1201 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1201 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1201 may be a combination of the above memories.

[0266] The processor 1202 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1202 is used to implement the above-mentioned shared bicycle illegal parking detection method when calling the computer program stored in the memory 1201.

[0267] The communication module 1203 is used to communicate with electronic devices and other servers.

[0268] The specific connection medium between the memory 1201, the communication module 1203 and the processor 1202 is not limited in the embodiment of the present application. Fig.12 In the embodiment, the memory 1201 and the processor 1202 are connected via a bus 1204. The bus 1204 is connected to the processor 1202 via a bus 1204. Fig.12 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1204 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.12 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0269] The memory 1201 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the shared bicycle illegal parking detection method of the embodiment of the present application. The processor 1202 is used to execute the above-mentioned shared bicycle illegal parking detection method. Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium, and the computer program product includes: computer program code, when the computer program code runs on the computer, the computer executes any of the shared bicycle illegal parking detection methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the shared bicycle illegal parking detection method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0270] Refer to the following Fig.13 1300 according to this embodiment of the present application is described. Fig.13 The computing device 1300 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0271] like Fig.13 The computing device 1300 is in the form of a general computing device. The components of the computing device 1300 may include, but are not limited to: at least one processing unit 1301, at least one storage unit 1302, and a bus 1303 connecting different system components (including the storage unit 1302 and the processing unit 1301).

[0272] Bus 1303 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0273] The storage unit 1302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0274] The storage unit 1302 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0275] The computing device 1300 may also communicate with one or more external devices 1304 (e.g., keyboards, pointing devices, etc.), one or more devices that enable a user to interact with the computing device 1300, and / or any device that enables the computing device 1300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via input / output (I / O) interface 1305. Furthermore, the computing device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1306. Fig.13 As shown, the network adapter 1306 communicates with other modules for the computing device 1300 via the bus 1303. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 1300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0276] The embodiment of the present application also provides a computer program product, and the method in the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.

[0277] A computer-readable storage medium can be implemented as a computer program product, that is, an embodiment of the present application also provides a computer-readable storage medium, which includes a computer program, and when the computer program is executed by a processor, it implements any of the above-mentioned shared bicycle illegal parking detection methods.

[0278] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0279] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0280] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0281] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0282] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0283] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting illegal parking of shared bicycles, characterized in that: The method comprises: Perform instance detection on each frame image in the video to be detected in turn, determine a target frame image containing a non-motor vehicle, and perform instance segmentation on the non-motor vehicle in the target frame image to obtain a body mask sub-image and a front mask sub-image of the non-motor vehicle; Inputting the vehicle body mask sub-image and the vehicle head mask sub-image into a pre-trained feature extraction network to obtain a first feature representing the style of the non-motor vehicle; The first feature is compared with at least one second feature in the target feature set; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, the non-motor vehicle is determined to be a shared bicycle corresponding to a specific style represented by the second feature; each of the second features is obtained by merging a body mask sub-image and a front mask sub-image of a shared bicycle of a specific style through the feature extraction network in advance and then performing feature extraction; the specific style includes a specific color, a specific structure and a specific shape; If it is determined that the shared bicycle is not in use by performing instance tracking on a plurality of consecutive image frames including the target image frame, illegal parking detection is performed on the shared bicycle in any one of the plurality of consecutive image frames.

2. The method according to claim 1, characterized in that By performing instance tracking on a plurality of consecutive frames of images including the target frame image, determining that the shared bicycle is in a non-use state includes: If it is determined that the shared bicycle is in a stationary state by performing instance tracking on the continuous multiple frames of images, performing instance detection on the last frame of the continuous multiple frames of images; If no human body is detected in the last frame of image, it is determined that the shared bicycle is in a non-use state; Alternatively, if a human body is detected in the last frame of the image, and the overlap ratio between each human body and the shared bicycle is not greater than a preset overlap threshold, it is determined that the shared bicycle is in a non-use state.

3. The method according to claim 2, characterized in that The method further comprises: If it is determined that the shared bicycle is in a non-stationary state by performing instance tracking on the continuous multiple frames of images, then no illegal parking detection is performed on the shared bicycle; Alternatively, if a human body is detected in the last frame image, and the overlap ratio between at least one human body and the shared bicycle is greater than the overlap threshold, it is determined that the shared bicycle is in use, and the shared bicycle is not detected for illegal parking.

4. The method according to any one of claims 1 to 3, characterized in that The illegal parking detection includes: at least one of parking area detection and vehicle head direction detection; the illegal parking detection of the shared bicycle in any one of the continuous multiple frames of images includes: Query the target object for the standard parking information pre-configured for the shared bicycle; If the standard parking information is found, the illegal parking detection is performed on the shared bicycle based on the standard parking information; If the standard parking information is not found, the standard parking information is determined by a pre-trained scene segmentation network, and the illegal parking detection is performed on the shared bicycle based on the standard parking information.

5. The method according to claim 4, characterized in that The illegal parking detection includes parking area detection, and the standard parking information includes a standard parking area corresponding to the specific style of the shared bicycle. Then, each frame image in the video to be detected is subjected to instance detection in turn to determine a target frame image containing a shared bicycle, including: Perform instance detection on each frame image in the video to be detected in turn, and determine a target frame image containing a shared bicycle and a coordinate frame of the shared bicycle; And, the detecting the illegal parking of the shared bicycle based on the standard parking information includes: The center coordinates of the coordinate frame are used as the position coordinates of the shared bicycle; If the location coordinates are within the standard parking area, it is determined that the shared bicycle is in the standard parking area; If the location coordinates are not within the standard parking area, it is determined that the shared bicycle is in an illegal parking area.

6. The method according to claim 5, characterized in that Before determining that the shared bicycle is in the standard parking area if the location coordinates are in the standard parking area, the method further includes: If the position coordinates are not in the motor vehicle lane area, determining whether the position coordinates are in the non-motor vehicle lane area; If the location coordinates are not in the non-motorized vehicle lane area, determining whether the location coordinates are in the sidewalk area; If the position coordinates are within the sidewalk area, it is determined whether the position coordinates are within the standard parking area of ​​the sidewalk area.

7. The method according to claim 4, characterized in that The illegal parking detection includes parking area detection, and the standard parking information includes a standard parking area. Then, the standard parking area is determined by a scene segmentation network, including: Input any frame of the image into a pre-trained scene segmentation network, and obtain a pedestrian mask sub-image according to the corresponding segmentation result; The standard parking area is generated according to the sidewalk mask subgraph, and the standard parking area represents a closed area formed by surrounding the sidewalk.

8. The method according to claim 4, characterized in that The illegal parking detection includes vehicle head direction detection, and the standard parking information includes a second direction vector. Then, performing the illegal parking detection on the shared bicycle based on the standard parking information includes: A first direction vector is constructed by taking the center point of the vehicle body mask sub-image as the starting point of the vector and taking the center point of the vehicle head mask sub-image as the end point of the vector; Determine the angle between the first direction vector and the second direction vector; the second direction vector represents the standard vehicle head direction; If the angle is less than a preset angle threshold, the front direction of the shared bicycle is determined to be in a standard direction; If the angle is not less than the preset angle threshold, it is determined that the front direction of the shared bicycle is in violation of the regulations.

9. The method according to claim 4, characterized in that The illegal parking detection includes vehicle head direction detection, and the standard parking information includes a second direction vector. Then, the second direction vector is determined by a scene segmentation network, including: Inputting any frame of the image into a pre-trained scene segmentation network, and obtaining a pedestrian mask sub-image and a road mask sub-image according to the corresponding segmentation results; The longest boundary line between the pedestrian mask sub-image and the road mask sub-image is used as the target boundary line; A direction vector that is perpendicular to the target boundary line and points to the lane mask sub-image is used as the second direction vector.

10. The method according to any one of claims 1 to 3, characterized in that: The illegal parking detection includes body posture detection, and the illegal parking detection of the shared bicycle in any one of the continuous multiple frames of images includes: Inputting the first feature into a pre-trained attribute classification network to obtain a classification value; If the classification value is greater than a preset classification threshold, it is determined that the shared bicycle is in a lying state; If the classification value is not greater than the preset classification threshold, it is determined that the shared bicycle is in an upright state.

11. A shared bicycle illegal parking detection device, characterized in that: The device comprises: An instance unit is used to perform instance detection on each frame image in the video to be detected in turn, determine a target frame image containing a non-motor vehicle, and perform instance segmentation on the non-motor vehicle in the target frame image to obtain a body mask sub-image and a front mask sub-image of the non-motor vehicle; A feature extraction unit, configured to input the vehicle body mask sub-image and the vehicle head mask sub-image into a pre-trained feature extraction network to obtain a first feature representing the style of the non-motor vehicle; A determination unit, configured to compare the first feature with at least one second feature in a target feature set; if there is a second feature in the target feature set whose feature similarity reaches a preset similarity threshold, then determining that the non-motor vehicle is a shared bicycle corresponding to a specific style represented by the second feature; each of the second features is obtained by merging a body mask sub-image and a front mask sub-image of a shared bicycle of a specific style through the feature extraction network in advance and then performing feature extraction; the specific style includes a specific color, a specific structure, and a specific shape; A detection unit is used to perform illegal parking detection on the shared bicycle in any frame of the continuous multiple frames if it is determined that the shared bicycle is in a non-use state by performing instance tracking on a continuous multiple frame image including the target frame image.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: It includes a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 10.

14. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1 to 10.

Images