Stand recognition and detection method and device and electronic equipment

Abstract

The invention provides a stall identification detection method and device and electronic equipment, and the method comprises the steps: obtaining a video frame collected for cruising a target region, importing the video frame into a detection model obtained through pre-training, and outputting the road information and stall information in the target region. And determining an illegal area and an allowable area in the target area based on the road information, and determining a stall area based on the stall information. And in combination with the stall area, the illegal area and the permissible area, detecting and judging whether the stall has an illegal stall setting phenomenon or not. In the scheme, on the basis of multi-task processing of the detection model, whether illegal stall exists or not can be accurately judged in combination with the road information and stall information obtained by the detection model. Based on the automatic detection mode of the image, the workload of the management personnel is reduced, and the method has the advantage of high real-time performance, thereby guaranteeing the management timeliness and improving the improvement effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a stall identification and detection method, device, and electronic equipment. Background Art

[0002] In the existing management method, urban management personnel need to conduct inspections frequently. In addition, the number of public places that need to be inspected is large, and the timeliness of management through manual inspections is also insufficient. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a stall identification and detection method, device and electronic equipment to accurately determine whether there are illegal stalls, ensure the timeliness of management, and improve the rectification effect.

[0004] In a first aspect, the present invention provides a stall identification and detection method, the method comprising:

[0005] Obtaining video frames collected by cruising the target area;

[0006] Importing the video frame into a pre-trained detection model to output road information and stall information within the target area;

[0007] determining an illegal area and a permitted area within the target area based on the road information, and determining a stall area based on the stall information;

[0008] In combination with the stall area, the illegal area and the permitted area, it is detected and determined whether there is any illegal stall setting.

[0009] In an optional embodiment, the step of obtaining video frames collected by cruising the target area includes:

[0010] Analyze the high-incidence areas of illegal street vendors based on the historical video frames obtained to determine the target areas;

[0011] The drone equipped with the camera device is controlled to cruise the target area to acquire video frames of the target area.

[0012] In an optional embodiment, the step of analyzing the high-incidence areas of illegal street vendors based on the obtained historical video frames to determine the target area includes:

[0013] obtaining a plurality of historical video frames within a study area and identifying illegal stalls in each of the historical video frames;

[0014] Clustering the study area based on the illegal stalls in each of the historical video frames to divide the study area into a plurality of sub-areas;

[0015] Target areas with a high incidence of illegal stalls are determined based on the frequency of illegal stalls in each of the sub-areas.

[0016] In an optional embodiment, the step of clustering the study area based on the illegal stalls in each of the historical video frames to divide the study area into a plurality of sub-areas includes:

[0017] Constructing global information of illegal stalls in the study area based on the illegal stalls in each of the historical video frames;

[0018] The study area is divided into sub-areas according to the global information of illegal stalls in the study area and a clustering algorithm is adopted.

[0019] In an optional embodiment, the step of controlling a drone equipped with a camera to cruise the target area to acquire video frames of the target area includes:

[0020] Setting at least one inspection point based on the area of ​​the target area;

[0021] Performing cruise path planning based on the starting point, the return point, and the at least one patrol point of the drone;

[0022] The drone equipped with the camera device is controlled to cruise the target area according to the cruise path to acquire video frames of the target area.

[0023] In an optional embodiment, the detection model includes a backbone network and an object detection head and a semantic segmentation head respectively connected to the backbone network;

[0024] The step of importing the video frame into a pre-trained detection model and outputting the road information and stall information in the target area comprises:

[0025] Importing the video frame into the backbone network of the pre-trained detection model to obtain feature information of the video frame, and inputting the feature information into the object detection head and the semantic segmentation head respectively;

[0026] Performing stall identification based on the feature information by the target detection head to output stall information within the target area;

[0027] Road segmentation processing is performed by the semantic segmentation head based on the feature information to output road information within the target area.

[0028] In an optional embodiment, the method further includes a step of pre-training to obtain the detection model, which step includes:

[0029] Obtaining a sample video frame, wherein the sample video frame is marked with a real stall label and a real road label;

[0030] Importing the sample video frame into the constructed deep learning model, and outputting feature information of the sample video frame through the backbone network of the deep learning model;

[0031] Importing the feature information into the target detection head and semantic segmentation head of the deep learning model respectively to obtain output stall information and output road information;

[0032] The object detection head is trained based on the constructed first sub-loss function, the semantic segmentation head is trained based on the constructed second sub-loss function, and the backbone network is trained based on the constructed total loss function until a preset stopping condition is met, thereby obtaining a detection model trained by the deep learning model;

[0033] The first sub-loss function is constructed by the real stall label and the output stall information, the second sub-loss function is constructed by the real road label and the output road information, and the total loss function is the superposition of the first sub-loss function and the second sub-loss function.

[0034] In an optional embodiment, the step of detecting and determining whether the stall has any illegal stall operation in combination with the stall area, the illegal area, and the permitted area includes:

[0035] Detect whether the stall area is within the allowed area and has no overlapping part with the illegal area. If it is within the allowed area and has no overlapping part with the illegal area, it is determined that the stall does not have an illegal stall phenomenon. Otherwise, it is determined that the stall has an illegal stall phenomenon.

[0036] In a second aspect, the present invention provides a stall identification and detection device, the device comprising:

[0037] An acquisition module is used to obtain video frames collected by cruising the target area;

[0038] A processing module, configured to import the video frame into a pre-trained detection model and output road information and stall information within the target area;

[0039] a determination module, configured to determine an illegal area and a permitted area within the target area based on the road information, and determine a stall area based on the stall information;

[0040] The detection module is used to detect and determine whether there is any illegal stall operation at the stall by combining the stall area, the illegal area and the allowed area.

[0041] In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement any of the methods described in the foregoing embodiments.

[0042] The present invention provides a stall identification and detection method, device and electronic equipment, which obtain video frames collected by patrolling a target area, import the video frames into a pre-trained detection model, and output road information and stall information in the target area. The illegal area and permitted area in the target area are determined based on the road information, and the stall area is determined based on the stall information. Combining the stall area, illegal area and permitted area, it is detected and judged whether there is illegal stall setting up. In this solution, multi-task processing based on the detection model can accurately judge whether there is illegal stall setting up based on the road information and stall information obtained by the detection model. The image-based automatic detection method reduces the workload of management personnel and has the advantage of high real-time performance, thereby ensuring the timeliness of management and improving the rectification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flowchart of a stall identification and detection method provided in an embodiment of the present invention;

[0045] Figure 2 for Figure 1 Flowchart of the sub-steps included in S11;

[0046] Figure 3 for Figure 2 Flowchart of the sub-steps included in S111;

[0047] Figure 4 for Figure 3 Flowchart of the sub-steps included in S1112;

[0048] Figure 5 for Figure 2 Flowchart of the sub-steps included in S112;

[0049] Figure 6 A flowchart of a model training method in the stall identification and detection method provided by an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the architecture of a deep learning model provided in an embodiment of the present invention;

[0051] Figure 8 for Figure 1 Flowchart of the sub-steps included in S13;

[0052] Figure 9 A functional module block diagram of a stall identification and detection device provided in an embodiment of the present invention;

[0053] Figure 10 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.

[0055] In order to maintain the city's image, existing illegal street vendors are mainly rectified by city management personnel through patrol inspections in different areas. However, although manual inspection of illegal street vendors can effectively curb such illegal street vendors, it requires manpower for detection and expulsion, and different personnel need to be assigned to patrol in different areas, which consumes a lot of manpower.

[0056] To quickly detect illegal street vendors, some use image detection algorithms to identify illegal street vendors. However, traditional image detection algorithms simply identify the stalls themselves and then need to combine the identified stalls with the identified objects to determine whether illegal activity is occurring. This lacks automation and does not completely address the issue of high labor consumption.

[0057] Based on these findings, the present invention provides a stall identification and detection method. This method, based on multi-task processing using a detection model, combines road and stall information obtained from the detection model to accurately determine whether illegal stalls are operating. This image-based automatic detection method reduces the workload of management personnel and offers the advantage of high real-time performance, ensuring timely management and improving the effectiveness of rectification.

[0058] See also Figure 1 , is a flow chart of a stall identification and detection method provided by an embodiment of the present invention. The stall identification and detection method can be applied to a stall identification and detection device. The stall identification and detection device can be loaded into an electronic device in the form of software. The electronic device can be a computer, server, laptop, etc. The specific implementation of the stall identification and detection method is as follows:

[0059] S11, obtaining video frames collected by cruising the target area.

[0060] S12, importing the video frame into a pre-trained detection model, and outputting the road information and stall information in the target area.

[0061] S13 , determining an illegal area and a permitted area within the target area based on the road information, and determining a stall area based on the stall information.

[0062] S14, combining the stall area, the illegal area and the permitted area, detecting and judging whether there is illegal stall setting.

[0063] In this embodiment, the target area can be an area determined by a certain method or an area designated manually. The target area is generally an area where there are many street vendors, such as an area with a large flow of people and a relatively wide venue.

[0064] The target area is captured by a camera, and thus a video frame of the target area can be obtained. The camera can be mounted on a drone, and the drone is controlled to cruise over the target area, so that the camera can capture and obtain the video frame of the target area.

[0065] Furthermore, in this embodiment, a pre-trained detection model is provided. This detection model can be pre-trained based on collected historical data. Given a video frame as input, this detection model can identify stall targets within the video frame and segment the roads within the video frame. Finally, it outputs road and stall information within the target area.

[0066] The road information may be a road boundary line, a road center line, etc. obtained by identifying road lines and road boundaries within the target area. The road area within the target area may be determined based on the obtained road information.

[0067] In real-world scenarios, street vendors may be permitted for short periods of time on certain roads. Therefore, not all road areas are prohibited from setting up street vendors. Based on this, in this embodiment, illegal and permitted areas within a target area can be determined based on road information. For example, areas near intersections within a road area are considered illegal areas, while wider areas farther from intersections within the road area are considered permitted areas.

[0068] Furthermore, stall information can be determined by identifying the pixel values ​​of pixels in the video frame within the target area and comparing foreground and background pixels. For example, background pixels are typically road pixels. If the foreground is a stall target, the pixel values ​​of the foreground and background pixels are generally significantly different. The resulting stall information includes the pixels corresponding to the identified stall target and the location information of each pixel.

[0069] The stall area can be delineated based on the position information of the pixel points corresponding to the stall target included in the stall information.

[0070] Based on this, the presence of illegal stalls can be determined by combining the stall area, illegal area, and permitted area detection. Specifically, the stall area is checked to see if it is within the permitted area and does not overlap with the illegal area. If it is within the permitted area and does not overlap with the illegal area, then the stall is determined to be free of illegal stalls. Otherwise, it is determined to be free of illegal stalls.

[0071] The stall identification and detection method provided in this embodiment uses a detection model to identify road and stall information. Combining these information accurately determines whether illegal stalls are operating. This image-based automatic detection method reduces the workload of management personnel and offers the advantage of high real-time performance, ensuring timely management and improving the effectiveness of rectification.

[0072] Considering that within the controlled areas, some areas have fewer stalls due to low foot traffic, while some areas have more foot traffic and therefore more stalls, which are also high-incidence areas for stalls, these high-incidence areas need to be focused on to ensure timely and effective rectification of illegal stalls.

[0073] Based on the above considerations, please refer to Figure 2 In this embodiment, the step of obtaining the video frames collected for cruising the target area can be achieved by:

[0074] S111, analyzing high-incidence areas of illegal street vendors based on the obtained historical video frames to determine a target area;

[0075] S112: Control a drone equipped with a camera to cruise the target area to acquire video frames of the target area.

[0076] In this embodiment, historical video frames within the study area within a historical period are obtained, wherein the study area can be an area of ​​a certain range within a city, which can be divided according to dimensions such as districts, streets, and communities in the city, or can also be divided according to the cruising range of a drone, etc., without any specific restrictions.

[0077] Study areas are generally large, with some areas experiencing high rates of illegal street vendors and others experiencing low rates. Therefore, historical video frames from the study area over a specific period can be analyzed to identify areas with high rates of illegal street vendors, which serve as target areas. Drones equipped with cameras can then be controlled to focus on these target areas, capturing video frames. This allows for subsequent identification and remediation of illegal street vendors within these areas.

[0078] See also Figure 3 In this embodiment, the above step of analyzing the high-incidence areas of illegal stalls based on the obtained historical video frames to determine the target area can be achieved by:

[0079] S1111, obtaining multiple historical video frames in the study area, and determining illegal stalls in each of the historical video frames;

[0080] S1112, clustering the study area based on the illegal stalls in each of the historical video frames to divide the study area into a plurality of sub-areas;

[0081] S1113, determining a target area with a high incidence of illegal stalls based on the frequency of illegal stalls in each of the sub-areas.

[0082] In this embodiment, the obtained historical video frames within the study area include historical video frames of various parts within the study area. Illegal stalls are marked in each historical video frame, and information about the illegal stalls can be recorded.

[0083] On this basis, clustering is performed on the study area. Specifically, a neighborhood radius and density threshold can be set, and clustering algorithms such as the DBSCAN algorithm can be used. This allows the study area to be divided into multiple sub-areas. The frequency of illegal stalls in each sub-area is then determined, and sub-areas with higher frequencies are designated as target areas.

[0084] See also Figure 4 In this embodiment, the step of clustering the study area based on the illegal stalls in each historical video frame to divide it into multiple sub-areas can be achieved by:

[0085] S11121: Construct global information of illegal stalls in the research area based on the illegal stalls in each of the historical video frames.

[0086] S11122: Divide the study area into sub-areas based on the global information of illegal stalls in the study area and using a clustering algorithm.

[0087] Because each historical video frame only captures a portion of the study area, it cannot fully reflect the overall status of illegal stalls within the study area. Therefore, in this embodiment, information about illegal stalls can be recorded for each historical video frame. Specifically, the longitude and latitude information and timestamp of the illegal stall can be obtained. Based on this information and timestamp, a three-dimensional feature vector is constructed. The elements of this three-dimensional feature vector include longitude, latitude, and a timestamp, which can be converted to a 24-hour format. This three-dimensional feature vector is used to characterize each illegal stall.

[0088] Because illegal stalls may appear multiple times at a given location over a period of time, the frequency of each illegal stall's appearance can be counted. For each illegal stall, if it appears once, the number of votes for its three-dimensional feature vector is incremented by 1. This allows us to determine the number of illegal stalls at each location within the study area, as well as their frequency of appearance. This allows us to construct a global picture of illegal stalls within the study area.

[0089] Based on the global information obtained, clustering algorithm is used to perform clustering processing, thereby dividing the study area into multiple sub-areas. Then, the target area is determined from the multiple sub-areas.

[0090] On this basis, see Figure 5 , the steps of controlling a drone equipped with a camera device to cruise the target area to acquire video frames of the target area can be achieved by:

[0091] S1121: Set at least one patrol point based on the area of ​​the target area.

[0092] S1122: Plan a cruise path based on the starting point, the return point, and the at least one patrol point of the drone.

[0093] S1123: Control the drone equipped with the camera device to cruise the target area according to the cruise path to acquire video frames of the target area.

[0094] When cruising, a drone must follow a cruising path. The starting and return points of the cruising path are fixed. For example, both the starting and return points can be the location of the drone's base station. Furthermore, the cruising path must include checkpoints, which are points above the target area.

[0095] Considering that different target areas have different areas, some target areas may be larger and some may be smaller. In order to ensure that the drone's patrol workload can be reduced while obtaining comprehensive image information of the target area, in this embodiment, at least one patrol point can be set based on the area of ​​the target area. For example, if the target area is small, one patrol point can be set, which can be the center point of the target area. If the target area is large, two or three or more patrol points can be set, and the multiple patrol points can be relatively evenly distributed within the target area.

[0096] After determining the starting point, return point, and patrol points, a cruise path can be planned based on the starting point, return point, and patrol points. For example, a greedy algorithm can be used to plan the cruise path to obtain the shortest cruise path starting from the starting point, passing through each patrol point, and returning to the return point.

[0097] In this way, the UAV is controlled to cruise along the cruise path, and during the cruise of the UAV, a camera device is used to capture video frames within the target area.

[0098] In this embodiment, clustering is performed based on historical video frames and a clustering algorithm to identify target areas with high incidence of illegal stalls. This allows for focused monitoring of illegal stalls in these areas. Furthermore, the locations of these areas can be updated based on the continuous updating of video frames, enabling efficient illegal stall detection.

[0099] After obtaining the video frame of the target area, the video frame is imported into the pre-trained detection model for recognition processing. Figure 6 , the following introduces the implementation method of the pre-trained detection model:

[0100] S21 , obtaining a sample video frame, wherein the sample video frame is marked with a real stall label and a real road label.

[0101] S22: Import the sample video frame into the constructed deep learning model, and output feature information of the sample video frame through the backbone network of the deep learning model.

[0102] S23, importing the feature information into the target detection head and semantic segmentation head of the deep learning model respectively to obtain output stall information and output road information.

[0103] S24, training the target detection head based on the constructed first sub-loss function, training the semantic segmentation head based on the constructed second sub-loss function, and training the backbone network based on the constructed total loss function, until the preset stopping condition is met, and obtaining a detection model trained by the deep learning model.

[0104] In this embodiment, drones equipped with cameras can be used to capture video frames from multiple angles in specific areas where illegal street vendors are being detected. Using these captured video frames from multiple angles for model training can improve the model's generalization capabilities across different viewpoints and scenarios.

[0105] From the captured video frames, we select those containing street vendors as sample frames. The stall objects and road information in these sample frames are labeled as true stall labels and true road labels. True stall labels are annotated using the same method as the stall object detection boxes, while true road labels are annotated using the same method as the road segmentation information. These sample frames serve as the core data foundation for subsequent model training, ensuring high model accuracy and adaptability.

[0106] In this embodiment, the deep learning model can use the YOLOv8 model as a basis, please refer to Figure 7 , the network architecture of the deep learning model is as follows Figure 7 As shown in , it includes a backbone network and an object detection head and a semantic segmentation head respectively connected to the backbone network. That is, the object detection head and the semantic segmentation head share a backbone network to extract common feature information.

[0107] After the backbone network processes the input sample video frames to obtain feature information, it is fed into different detection head networks, including the object detection head and the semantic segmentation head. Each detection head network processes the feature information to suit its respective application needs. In this way, the backbone network in the shared model serves as a unified feature extractor for both object detection and road segmentation tasks, reducing redundant computation and enhancing semantic synergy between the two tasks.

[0108] During the training process, the model is trained based on the constructed loss function. The object detection head and semantic segmentation head are trained based on the first and second sub-loss functions, respectively, while the backbone network is trained based on the total loss function.

[0109] Among them, the first sub-loss function is constructed by the real stall label and the output stall information, the second sub-loss function is constructed by the real road label and the output road information, and the total loss function is the superposition of the first sub-loss function and the second sub-loss function.

[0110] In this way, the gradient of the backbone network is backpropagated by the sum of the losses of the target detection head and the segmentation semantic head, while the gradient updates of the target detection head and the segmentation semantic head are determined only by their respective losses, the gradient update of the semantic segmentation head is determined by the loss of semantic segmentation, and the gradient update of the target detection head is determined by the loss of target detection.

[0111] In this way, gradient decoupling optimization is implemented between branches to prevent the main task of target detection from affecting the learning stability of the segmentation subtask.

[0112] Based on the pre-trained detection model, it can be deployed on the actual application platform. To meet the actual application requirements, the trained detection model needs to undergo format conversion and performance optimization to adapt to the target deployment platform. For example, when the detection model is deployed on the server, it uses the Triton Inference Server for centralized scheduling and accelerated inference, which requires completing the conversion process from the training format to an efficient inference format.

[0113] Specifically, the .pth weight files obtained from PyTorch training are first exported to the intermediate ONNX format (.onnx) to achieve compatibility between frameworks and cross-platform deployment. Using the NVIDIA TensorRT toolchain, the ONNX model is further converted into a highly optimized TensorRT Engine file (.engine) to fully utilize the GPU for low-latency inference.

[0114] After the deployment is completed, in the actual application stage, the video frames of the target area are obtained through the above method, and the video frames are imported into the detection model for recognition and detection processing to obtain the road information and stall information in the target area. Figure 8 Specifically, this can be achieved by:

[0115] S131, importing the video frame into the backbone network of the pre-trained detection model to obtain feature information of the video frame, and inputting the feature information into the target detection head and the semantic segmentation head respectively.

[0116] S132: Perform stall recognition based on the feature information by using the target detection head to output stall information within the target area.

[0117] S133: Perform road segmentation processing based on the feature information through the semantic segmentation head to output road information within the target area.

[0118] The detection model is trained using a deep learning model and therefore also includes a backbone network, an object detection head, and a semantic segmentation head. After the backbone network extracts features from video frames, the object detection head detects and outputs stall information based on these features. Simultaneously, the semantic segmentation head detects and outputs road information based on these features.

[0119] Finally, illegal and permitted areas can be determined based on road information, and stall areas can be determined based on stall information. By comparing the stall area with the illegal and permitted areas, it can be determined whether the stall is illegally set up.

[0120] If it is determined that the stall is set up illegally, the management personnel can be notified in time to manage it, or broadcasting equipment can be set up in each area to send prompt information to the broadcasting equipment in the area where the illegal stall is set up, so as to issue a prompt voice through the broadcasting equipment to remind the illegal stall owners to prohibit the stall setting behavior.

[0121] In addition, relevant video frames may be saved, and information such as the timestamp, location information, and detection frame of the stall area in which the stall information is detected may also be saved.

[0122] The video frames of illegal street stalls detected in the actual application stage can be used as the basis data for updating areas with high incidence of illegal street stalls. For example, the video frames of illegal street stalls can be added to the data set of detection areas with high incidence of illegal street stalls, so as to timely update the information of areas with high incidence of illegal street stalls.

[0123] The stall recognition and detection method provided in this embodiment automatically detects road and stall information through video acquisition and model detection. Furthermore, a multi-task detection model for road segmentation and stall detection is trained to meet the real-time requirements of video stream recognition in practical applications.

[0124] In addition, clustering is used through existing historical video frames to determine the areas with high incidence of illegal stalls, and then key patrols are deployed in these areas, which can effectively achieve control and avoid the workload of patrols.

[0125] The technology of the present invention is applied to the identification of illegal street stalls, which can effectively solve the needs of users and supervise illegal street stalls. At the same time, it can facilitate inspections by patrol personnel and can independently plan routes. If there are similar related scenarios in the future, this method can also be used as a reference for design, which has certain technical reference value.

[0126] In order to execute the corresponding steps in the above stall identification and detection method embodiment and various possible methods, an implementation method of a stall identification and detection device is given below. Optionally, the stall identification and detection device can be loaded into an electronic device, such as a computer device, a server, etc.

[0127] Further, see Figure 9 , Figure 9 This is a functional module diagram of a stall identification and detection device provided in an embodiment of the present invention. It should be noted that the basic principles and technical effects of the stall identification and detection device provided in this embodiment are the same as those of the corresponding method embodiments described above. For the sake of simplicity, any parts not mentioned in this embodiment can be referred to the corresponding contents of the corresponding method embodiments described above. The stall identification and detection device includes:

[0128] An acquisition module is used to obtain video frames collected by cruising the target area;

[0129] A processing module, configured to import the video frame into a pre-trained detection model and output road information and stall information within the target area;

[0130] a determination module, configured to determine an illegal area and a permitted area within the target area based on the road information, and determine a stall area based on the stall information;

[0131] The detection module is used to detect and determine whether there is any illegal stall operation at the stall by combining the stall area, the illegal area and the allowed area.

[0132] As a possible implementation, the acquisition module may be used to obtain a video frame of the target area in the following manner:

[0133] Analyze the high-incidence areas of illegal street vendors based on the historical video frames obtained to determine the target areas;

[0134] The drone equipped with the camera device is controlled to cruise the target area to acquire video frames of the target area.

[0135] As a possible implementation, the acquisition module may be used to determine the target area in the following manner:

[0136] obtaining a plurality of historical video frames within a study area and identifying illegal stalls in each of the historical video frames;

[0137] Clustering the study area based on the illegal stalls in each of the historical video frames to divide the study area into a plurality of sub-areas;

[0138] Target areas with a high incidence of illegal stalls are determined based on the frequency of illegal stalls in each of the sub-areas.

[0139] As a possible implementation, the acquisition module may be used to divide the sub-regions in the following manner:

[0140] Constructing global information of illegal stalls in the study area based on the illegal stalls in each of the historical video frames;

[0141] The study area is divided into sub-areas according to the global information of illegal stalls in the study area and a clustering algorithm is adopted.

[0142] As a possible implementation, the acquisition module can control the drone in the following ways:

[0143] Setting at least one inspection point based on the area of ​​the target area;

[0144] Performing cruise path planning based on the starting point, the return point, and the at least one patrol point of the drone;

[0145] The drone equipped with the camera device is controlled to cruise the target area according to the cruise path to acquire video frames of the target area.

[0146] As a possible implementation, the processing module can be used to obtain road information and stall information in the following manner:

[0147] Importing the video frame into the backbone network of the pre-trained detection model to obtain feature information of the video frame, and inputting the feature information into the object detection head and the semantic segmentation head respectively;

[0148] Performing stall identification based on the feature information by the target detection head to output stall information within the target area;

[0149] Road segmentation processing is performed by the semantic segmentation head based on the feature information to output road information within the target area.

[0150] As a possible implementation, the stall identification and detection device further includes a detection module for pre-training a detection model, and the detection module is specifically used to:

[0151] Obtaining a sample video frame, wherein the sample video frame is marked with a real stall label and a real road label;

[0152] Importing the sample video frame into the constructed deep learning model, and outputting feature information of the sample video frame through the backbone network of the deep learning model;

[0153] Importing the feature information into the target detection head and semantic segmentation head of the deep learning model respectively to obtain output stall information and output road information;

[0154] The object detection head is trained based on the constructed first sub-loss function, the semantic segmentation head is trained based on the constructed second sub-loss function, and the backbone network is trained based on the constructed total loss function until a preset stopping condition is met, thereby obtaining a detection model trained by the deep learning model;

[0155] The first sub-loss function is constructed by the real stall label and the output stall information, the second sub-loss function is constructed by the real road label and the output road information, and the total loss function is the superposition of the first sub-loss function and the second sub-loss function.

[0156] As a possible implementation, the above detection module can be specifically used to:

[0157] Detect whether the stall area is within the allowed area and has no overlapping part with the illegal area. If it is within the allowed area and has no overlapping part with the illegal area, it is determined that the stall does not have an illegal stall phenomenon. Otherwise, it is determined that the stall has an illegal stall phenomenon.

[0158] See also Figure 10 , an electronic device provided in an embodiment of the present invention, and the stall identification and detection method provided in an embodiment of the present invention can be applied to this electronic device. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected to each other directly or indirectly to enable data transmission or exchange. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0159] Memory is used to store programs or data. Memory can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM).

[0160] The processor is used to read / write data or programs stored in the memory and execute corresponding functions.

[0161] The communication module is used to establish a communication connection between the electronic device and other communication terminals through the network, and is used to send and receive data through the network.

[0162] It should be understood that Figure 10 The structure shown is only a schematic diagram of the structure of the electronic device. The electronic device may also include Figure 10 More or fewer components than shown, or with Figure 10 Different configurations shown. Figure 10 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0163] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the stall identification and detection method provided in the above embodiment is implemented.

[0164] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is executed, the aforementioned stall identification and detection method can be implemented. The processes involved in executing the computer-readable storage medium and its executable instructions can be found in the description of the aforementioned method embodiments and will not be further elaborated here.

[0165] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, the indirect coupling or communication connection of the device or unit may be electrical, mechanical or other forms.

[0166] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] Furthermore, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0168] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0169] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0170] The foregoing description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A stall identification and detection method, characterized in that: The method comprises: Obtaining video frames collected by cruising the target area; Importing the video frame into a pre-trained detection model to output road information and stall information within the target area; determining an illegal area and a permitted area within the target area based on the road information, and determining a stall area based on the stall information; In combination with the stall area, the illegal area and the permitted area, it is detected and determined whether there is any illegal stall setting.

2. The stall identification and detection method according to claim 1, characterized in that: The step of obtaining video frames collected by cruising the target area includes: Analyze the high-incidence areas of illegal street vendors based on the historical video frames obtained to determine the target areas; The drone equipped with the camera device is controlled to cruise the target area to acquire video frames of the target area.

3. The stall identification and detection method according to claim 2, characterized in that: The step of analyzing the high-incidence areas of illegal street vendors based on the obtained historical video frames to determine the target area includes: obtaining a plurality of historical video frames within a study area and identifying illegal stalls in each of the historical video frames; Clustering the study area based on the illegal stalls in each of the historical video frames to divide the study area into a plurality of sub-areas; Target areas with a high incidence of illegal stalls are determined based on the frequency of illegal stalls in each of the sub-areas.

4. The stall identification and detection method according to claim 3, characterized in that: The step of clustering the study area based on the illegal stalls in each of the historical video frames to divide it into a plurality of sub-areas includes: Constructing global information of illegal stalls in the study area based on the illegal stalls in each of the historical video frames; The study area is divided into sub-areas according to the global information of illegal stalls in the study area and a clustering algorithm is adopted.

5. The stall identification and detection method according to claim 2, characterized in that: The step of controlling a drone equipped with a camera to cruise the target area to acquire video frames of the target area includes: Setting at least one inspection point based on the area of ​​the target area; Planning a cruise path based on the starting point, the return point, and the at least one patrol point of the drone; The drone equipped with the camera device is controlled to cruise the target area according to the cruise path to acquire video frames of the target area.

6. The stall identification and detection method according to claim 1, characterized in that: The detection model includes a backbone network and a target detection head and a semantic segmentation head respectively connected to the backbone network; The step of importing the video frame into a pre-trained detection model and outputting the road information and stall information in the target area comprises: Importing the video frame into the backbone network of the pre-trained detection model to obtain feature information of the video frame, and inputting the feature information into the object detection head and the semantic segmentation head respectively; Performing stall identification based on the feature information by the target detection head to output stall information within the target area; Road segmentation processing is performed by the semantic segmentation head based on the feature information to output road information within the target area.

7. The stall identification and detection method according to claim 1, characterized in that: The method further includes a step of pre-training to obtain the detection model, which step includes: Obtaining a sample video frame, wherein the sample video frame is marked with a real stall label and a real road label; Importing the sample video frame into the constructed deep learning model, and outputting feature information of the sample video frame through the backbone network of the deep learning model; Importing the feature information into the target detection head and semantic segmentation head of the deep learning model respectively to obtain output stall information and output road information; The object detection head is trained based on the constructed first sub-loss function, the semantic segmentation head is trained based on the constructed second sub-loss function, and the backbone network is trained based on the constructed total loss function until a preset stopping condition is met, thereby obtaining a detection model trained by the deep learning model; The first sub-loss function is constructed by the real stall label and the output stall information, the second sub-loss function is constructed by the real road label and the output road information, and the total loss function is the superposition of the first sub-loss function and the second sub-loss function.

8. The stall identification and detection method according to claim 1, characterized in that: The step of detecting and judging whether the stall has illegal stall operation in combination with the stall area, the illegal area and the permitted area includes: Detect whether the stall area is within the allowed area and has no overlapping part with the illegal area. If it is within the allowed area and has no overlapping part with the illegal area, it is determined that the stall does not have an illegal stall phenomenon. Otherwise, it is determined that the stall has an illegal stall phenomenon.

9. A stall identification and detection device, characterized in that: The device comprises: An acquisition module is used to obtain video frames collected by cruising the target area; A processing module, configured to import the video frame into a pre-trained detection model and output road information and stall information within the target area; a determination module, configured to determine an illegal area and a permitted area within the target area based on the road information, and determine a stall area based on the stall information; The detection module is used to detect and determine whether there is any illegal stall setting up at the stall by combining the stall area, the illegal area and the allowed area.

10. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the method according to any one of claims 1 to 8.

Images