Method and apparatus for detecting violations

Abstract

A method for detecting a violation, comprising: acquiring a plurality of image frames corresponding to a to-be-detected region (301); performing scene semantic segmentation on the plurality of image frames to obtain a prohibited violation region in the plurality of image frames (302); detecting the plurality of image frames by using a rotating target detection network (303), the rotating target detection network being configured to detect whether the image frames include a target detection object; and in a case where at least one of the plurality of image frames includes the target detection object, determining that the to-be-detected region has a violation according to the target detection object and a prohibited violation region corresponding to the target detection object (304).

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of intelligent detection, and in particular, to a method and device for detecting illegal behavior. BACKGROUND

[0002] In order to ensure that the environment to be detected (for example, a residential area, a park, an office building, a shopping mall, etc.) is clean and tidy, the illegal behavior in the environment to be detected can be detected. For example, the behavior of illegally posting advertisements and illegally airing clothes in the environment to be detected can be detected. SUMMARY

[0003] In one aspect, a method for detecting illegal behavior is provided. The method includes: first, obtaining a plurality of image frames corresponding to a region to be detected; then, performing scene semantic segmentation on the plurality of image frames to obtain an illegal area in the plurality of image frames; then, detecting the plurality of image frames using a rotating target detection network, the rotating target detection network being configured to detect whether the image frames include a target detection object; and finally, in a case where at least one of the plurality of image frames includes the target detection object, determining that the region to be detected has illegal behavior based on the target detection object and an illegal area corresponding to the target detection object.

[0004] In some embodiments, the target detection object includes at least one of an advertisement, clothes, or a pedestrian.

[0005] In some embodiments, the determining that the region to be detected has illegal behavior based on the target detection object and the illegal area corresponding to the target detection object includes: determining a ratio of an intersection area of a detection frame region of the target detection object and the illegal area corresponding to the target detection object; determining that the target detection object is in the illegal area when the ratio of the intersection area and a detection frame area of the target detection object is greater than or equal to a first preset threshold; and determining that the region to be detected has illegal behavior based on a detection state count value of the target detection object.

[0006] For example, when the target detection object includes an advertisement, the determining that the region to be detected has illegal behavior based on the detection state count value of the target detection object includes: increasing the detection state count value of the target detection object by one when a coordinate point displacement of the target detection object in adjacent two of the at least one image frame is less than a second preset threshold; and determining that the region to be detected has illegal behavior when the detection state count value of the target detection object is greater than or equal to a third preset threshold.

[0007] Exemplarily, when the target detection object includes clothes and pedestrians, the determining that the to-be-detected region has the illegal behavior according to the detection state count value of the target detection object includes: when the center point of the detection frame of the clothes is outside the detection frame region of at least one pedestrian, adding one to the detection state count value of the clothes; and when the detection state count value of the clothes is greater than or equal to a fourth preset threshold, determining that the to-be-detected region has the illegal behavior.

[0008] In some embodiments, the area of the detection frame of the at least one pedestrian is greater than or equal to a fifth preset threshold, and the center point of the detection frame of the at least one pedestrian is in the illegal behavior prohibited region corresponding to the target detection object.

[0009] In some embodiments, the detection method of the illegal behavior further includes: deleting the detection frame in the detection frame of the pedestrian, which has an area less than the fifth preset threshold or a center point outside the illegal behavior prohibited region.

[0010] In some embodiments, the rotating target detection network includes a Backbone network, a Neck network, and a Head network, and an output of the Head network includes a predicted rotation angle of the detection frame.

[0011] In some embodiments, the detecting, by the rotating target detection network, the plurality of image frames includes: pre-processing the image frames, extracting feature data of the image frames through the Backbone network and the Neck network, inputting the feature data of the image frames into the Head network to obtain a detection result of the target detection object, and the detection result of the target detection object includes at least one of a detection frame of the target detection object, a region coordinate of the detection frame, a predicted rotation angle of the detection frame, or a category of the target detection object.

[0012] In some embodiments, the detection method of the illegal behavior further includes: when the alarm indication indicates that no alarm has occurred, reporting the alarm information and changing the alarm indication; and when the alarm indication indicates that the alarm has occurred and a time from a last alarm is greater than a preset time, reporting the alarm information.

[0013] In another aspect, a device for detecting an illegal behavior is provided, including a processor and a transceiver. The transceiver is configured to obtain a plurality of image frames corresponding to a to-be-detected region. The processor is configured to: perform scene semantic segmentation on the plurality of image frames to obtain an illegal behavior prohibited region in the plurality of image frames; detect the plurality of image frames by using a rotating target detection network, the rotating target detection network being used to detect whether the image frames include a target detection object; and in a case where at least one image frame in the plurality of image frames includes the target detection object, determine that the to-be-detected region has the illegal behavior according to the target detection object and the illegal behavior prohibited region corresponding to the target detection object.

[0014] In some embodiments, the target detection object includes at least one of an advertisement, a piece of clothing, or a pedestrian.

[0015] In some embodiments, the processor is configured to: determine a ratio of an intersection area of the bounding box region of the target detection object and the prohibited violation region to an area of the bounding box of the target detection object; determine that the target detection object is in the prohibited violation region when the ratio of the intersection area to the area of the bounding box of the target detection object is greater than or equal to a first preset threshold; and determine that the violation behavior exists in the to-be-detected region according to a detection state count value of the target detection object.

[0016] In some embodiments, the processor is configured to: increase the detection state count value of the target detection object by one when a coordinate point of the target detection object in at least one of the image frames moves less than a second preset threshold in adjacent image frames; and determine that the violation behavior exists in the to-be-detected region when the detection state count value of the target detection object is greater than or equal to a third preset threshold.

[0017] In some embodiments, the processor is configured to: increase the detection state count value of the piece of clothing by one when a center point of the bounding box of the piece of clothing is outside the bounding box region of the at least one pedestrian; and determine that the violation behavior exists in the to-be-detected region when the detection state count value of the piece of clothing is greater than or equal to a fourth preset threshold.

[0018] In some embodiments, an area of the bounding box of the at least one pedestrian is greater than or equal to a fifth preset threshold, and a center point of the bounding box of the at least one pedestrian is in the prohibited violation region corresponding to the target detection object.

[0019] In some embodiments, the processor is further configured to: delete the bounding box of the pedestrian whose area is less than the fifth preset threshold or whose center point is outside the prohibited violation region.

[0020] In some embodiments, the target detection network includes a Backbone network, a Neck network, and a Head network, and an output of the Head network includes a predicted rotation angle of the bounding box.

[0021] In some embodiments, the processor is configured to: pre-process the image frame, extract feature data of the image frame through the Backbone network and the Neck network; input the feature data of the image frame into the Head network to obtain a detection result of the target detection object; and the detection result of the target detection object includes at least one of a bounding box of the target detection object, a region coordinate of the bounding box, a predicted rotation angle of the bounding box, or a category of the target detection object.

[0022] In some embodiments, the transceiver is further configured to: when the alarm identifier indicates that no alarm has occurred, send the alarm information and change the alarm identifier; and when the alarm identifier indicates that an alarm has occurred and the time since the last alarm is greater than a preset time, send the alarm information.

[0023] In another aspect, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium storing computer program instructions, which, when executed by a detection device, implement the method for detecting a violation behavior according to any one of the above embodiments.

[0024] In yet another aspect, a computer program product stored on a non-transitory computer-readable storage medium is provided, the computer program product comprising computer program instructions for causing a computer to execute the method for detecting a violation behavior according to any one of the above embodiments.

[0025] In yet another aspect, a computer program is provided. When the computer program is executed on a computer, the computer program causes the computer to execute the method for detecting a violation behavior according to any one of the above embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present disclosure, the following will briefly introduce the drawings needed to be used in some embodiments of the present disclosure. Obviously, the drawings described in the following description are only the drawings of some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art according to these drawings. In addition, the drawings described in the following description can be regarded as schematic diagrams, and are not limited to the actual size, actual flow, actual timing, etc. of the product, method, signal, etc. involved in the embodiments of the present disclosure.

[0027] Figure 1 a block diagram of a violation behavior detection system according to some embodiments;

[0028] Figure 2 a hardware device diagram of a violation behavior detection system according to some embodiments;

[0029] Figure 3 a flowchart of a violation behavior detection method according to some embodiments;

[0030] Figure 4A an architecture diagram of a rotating target detection network according to some embodiments;

[0031] Figure 4B a schematic diagram of a Head network in a rotating target detection network according to some embodiments;

[0032] Figure 5 an application scenario diagram of a rotating target detection network according to some embodiments;

[0033] Figure 6 Flow chart of a method for detecting another violation according to some embodiments;

[0034] Figure 7 Flow chart of a method for detecting another violation according to some embodiments;

[0035] Figure 8 Flow chart of a method for detecting another violation according to some embodiments;

[0036] Figure 9 Flow chart of a method for detecting another violation according to some embodiments;

[0037] Figure 10 Flow chart of a method for detecting another violation according to some embodiments;

[0038] Figure 11 Flow chart of a method for detecting a violation of posting an advertisement according to some embodiments;

[0039] Figure 12 Flow chart of a method for detecting a violation of hanging laundry according to some embodiments;

[0040] Figure 13 Structure diagram of a violation detection device according to some embodiments. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments provided by the present disclosure, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present disclosure.

[0042] Unless otherwise required by context, the term "comprise" and variations of the term such as "comprises" and "comprising" throughout the specification and claims, are to be construed as open-ended, that is, as "including, but not limited to." In describing the disclosure, the terms "one embodiment," "some embodiments,” “exemplary embodiments,” “example,” “specific example” or “some examples” are used to indicate that the particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the disclosure. Such terms are not necessarily used consistently in the specification and claims, and are not necessarily used to indicate the same embodiment or example. Furthermore, the described features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0043] Hereinafter, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the disclosure, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0044] In order to ensure that the environment to be detected (for example, a residential area, a park, an office building, a shopping mall, etc.) is clean and tidy, the violation behavior in the environment to be detected can be detected. The violation behavior includes, but is not limited to, pasting advertisements, airing clothes, etc. in the forbidden area (also referred to as the prohibited violation area) of the environment to be detected.

[0045] Generally, there are two ways to detect the violation behavior, one is to detect the violation behavior in the environment to be detected through artificial inspection. However, the detection method of artificial inspection has high cost and low management efficiency. The other is to detect the violation behavior in the environment to be detected through a monitoring device. However, due to the randomness of some violation behaviors, when the monitoring device is used to detect the violation behavior in the environment to be detected, a large amount of noise will be introduced, and the violation behavior cannot be accurately detected. For example, the violation-pasted advertisement often appears to be inclined due to the randomness of manual pasting, and if a horizontal bounding box is used for target detection, a large amount of background noise will be introduced, and it is difficult to accurately detect the inclined advertisement. For another example, the clothes aired in the forbidden area will flutter in the wind, and if a horizontal bounding box is used for target detection, a large amount of background noise will be introduced, and it is difficult to accurately detect the inclined clothes.

[0046] Furthermore, when using surveillance equipment to detect violations, restricted areas are typically pre-defined manually within the area captured by the equipment. When the camera angle changes, these restricted areas must be manually redefined before any violations within those areas can be detected, resulting in a lack of flexibility.

[0047] Therefore, some embodiments of this disclosure provide a method for detecting violations. By constructing a rotating target detection network, violations within the detection area can be accurately detected, improving management efficiency and reducing the cost of detecting violations. Furthermore, when the shooting angle of the monitoring equipment changes, the image captured by the monitoring equipment can be re-segmented into scene semantics to obtain the corresponding no-go zones. Therefore, there is no need for manual re-marking of no-go zones, which improves the flexibility of detection.

[0048] The violation detection methods provided in some embodiments of this disclosure can be applied to... Figure 1 The detection system shown. For example... Figure 1 As shown, the detection system includes a data acquisition module 10, a scene semantic segmentation module 11, a detection and analysis module 12, a data storage module 13, and a transmission module 14.

[0049] The data acquisition module 10 is used to acquire image frames and send them to the scene semantic segmentation module 11 and the detection and analysis module 12, respectively. The data acquisition module 10 can extract image frames based on the video stream captured by the monitoring device (e.g., a camera) within the area to be detected and the camera information. The area to be detected can be an area captured by at least one monitoring device in the environment to be detected.

[0050] The scene semantic segmentation module 11 is used to perform scene semantic segmentation on the image frames acquired by the data acquisition module 10, obtain the scene semantic segmentation result, and store the scene semantic segmentation result in the data storage module 13. For example, the scene semantic segmentation module 11 can segment uncountable background categories such as grass, road, sky, water, and wall in the image frame to obtain the scene semantic segmentation result.

[0051] In some embodiments, after the camera angle is fixed, the scene semantic segmentation module 11 can perform scene semantic segmentation on the area to be detected captured by the camera, and the scene semantic segmentation result can be stored in the data storage module 13. It is understood that when the camera angle does not change, the scene semantic segmentation results of multiple frames of images acquired by the data acquisition module 10 are the same. Therefore, when the camera angle remains unchanged, it is not necessary to perform scene semantic segmentation on each frame of the image again; the scene semantic segmentation result stored in the data storage module 13 can be used directly to further determine whether there is any violation in the area to be detected.

[0052] In some embodiments, when the shooting angle of the camera changes, the detection system can re-perform scene semantic segmentation on the new scene shot by the camera, and store the re-segmented scene semantic segmentation result in the storage module 13. It can be understood that, compared with manually re-calibrating the forbidden area, the detection system provided by the present disclosure can obtain the corresponding forbidden area by re-performing scene semantic segmentation on the image shot by the camera through the scene semantic segmentation module 11 when the shooting angle of the camera changes, so that it is not necessary to manually re-calibrate the forbidden area, and the flexibility of detection can be improved.

[0053] In some embodiments, other modules (for example, the detection and analysis module 12) in the detection system can demarcate the forbidden area according to the scene semantic segmentation result. The forbidden areas demarcated for different violation behaviors can be partially the same, all the same, or all different. For example, the detection and analysis module 12 can demarcate part or all of the scene semantic segmentation result obtained by the scene semantic segmentation module 11 as the forbidden area. For example, taking the scene semantic segmentation result obtained by the scene semantic segmentation module 11 as an example, which includes grassland, road, sky, water surface, and wall surface, the wall surface, grassland, and road can be demarcated as the forbidden area.

[0054] The detection and analysis module 12 is configured to detect and track the target detection object in the image frame, and when the image frame includes the target detection object, determine whether the violation behavior exists in the detection area according to the scene semantic segmentation result. The target detection object includes, but is not limited to, at least one of the detection objects such as pedestrians, clothes, and advertisements. Figure 1 As shown in FIG. 6, when the detection system is used to detect the violation of pasting advertisements and the violation of drying clothes, the detection and analysis module 12 is configured to perform pedestrian detection, clothes detection, and advertisement detection on the image frame sent by the data acquisition module 10.

[0055] For example, when the detection and analysis module 12 determines that the image frame acquired by the data acquisition module 10 includes the target detection object, the forbidden area can be demarcated according to the scene semantic segmentation result stored in the data storage module 13, and whether the violation behavior exists in the detection area can be determined according to the detection frame of the target detection object and the forbidden area corresponding to the target detection object.

[0056] In some embodiments, the detection and analysis module 12 can detect the target detection object in the image frame by rotating the target detection network, obtain a detection frame closer to the target detection object, and reduce redundant information in network learning and strategy making. The target detection object in the image frame can be accurately detected when the target detection object is in a tilted state.

[0057] In some embodiments, the detection and analysis module 12 is further configured to store the detection result of the target detection object in the data storage module 13.

[0058] The data storage module 13 is used to store the scene semantic segmentation results obtained by the scene semantic segmentation module 11 and the detection results of the target objects detected by the detection and analysis module 12.

[0059] The sending module 14 is used to send alarm or prompt information to the property management center when the detection and analysis module 12 determines that there is a violation in the area to be detected.

[0060] The violation detection methods provided in some embodiments of this disclosure can be applied to violation detection devices. Figure 2 This disclosure provides a hardware device diagram of a violation detection system according to some embodiments, such as... Figure 2 As shown, the violation detection device 20 includes at least one processor 21, a memory 22, a transceiver 23, and a communication bus 24.

[0061] The following is combined with Figure 2 The following is a detailed description of each component of the violation detection device 20:

[0062] Processor 21 is the control center of violation detection device 20. It can be a single processor or a collective term for multiple processing elements. For example, processor 21 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this disclosure, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0063] The processor 21 can perform various functions of the violation detection device 20 by running or executing software programs stored in the memory 22 and calling data stored in the memory 22.

[0064] In a specific implementation, as one example, the processor 21 may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 are shown in the diagram.

[0065] The memory 22 can be a read-only memory (ROM) or other type of static storage communication device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage communication device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, a magneto-optical disk storage, a magnetic disk storage or other magnetic storage communication device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory 22 can exist independently and be connected to the processor 21 through the communication bus 24. The memory 22 can also be integrated with the processor 21.

[0066] The memory 22 is configured to store a software program for implementing the scheme of the present disclosure and is controlled by the processor 21 to execute.

[0067] The transceiver 23 is configured to communicate with other communication devices. Of course, the transceiver 23 can also be used to communicate with a communication network, such as an Ethernet, a radio access network (RAN), a Wireless Local Area Networks (WLAN), etc. The transceiver 23 can include a receiving unit to implement a receiving function and a sending unit to implement a sending function.

[0068] The communication bus 24 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0069] Figure 2The structure of the violation detection apparatus 20 shown in the figure does not constitute a limitation on the violation detection apparatus 20, and the violation detection apparatus 20 can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0070] Some embodiments of the present disclosure provide a method for detecting a violation, as shown in the figure, the method comprises steps 301-304. Figure 3

[0071] Step 301, obtaining a plurality of image frames corresponding to a to-be-detected region.

[0072] The to-be-detected region is a region photographed by a monitoring device (for example, a camera) in a to-be-detected environment. The to-be-detected environment includes but is not limited to a residential area, a park, an office building, a shopping mall, etc. The to-be-detected environment can include one or more to-be-detected regions, and each to-be-detected region is a region photographed by a monitoring device in the to-be-detected environment.

[0073] In some embodiments, obtaining a plurality of image frames corresponding to a to-be-detected region comprises: extracting a plurality of image frames according to a video stream of the to-be-detected region photographed by the camera and camera information.

[0074] In some embodiments, the plurality of image frames can be photographed by the same camera at the same angle, or can be photographed by the same camera at different angles, and the present disclosure does not limit this. The camera can be any camera in the to-be-detected environment.

[0075] In some embodiments, step 301 can be performed by a data acquisition module 10 in the detection system shown in the figure. Figure 1

[0076] Step 302, performing scene semantic segmentation on the plurality of image frames to obtain a prohibited violation region in the plurality of image frames.

[0077] In some embodiments, a semantic segmentation model can be used to perform scene semantic segmentation on the image frames to obtain a scene semantic segmentation result. The semantic segmentation model can use SwinTransformer as the backbone network and UperNet as the overall network framework.

[0078] For example, taking a park as the to-be-detected environment, a scene semantic segmentation model of UperNet+SwinTransformer can be used to segment the non-countable background categories such as grass, road, sky, water surface, and wall surface in the image frames photographed by the cameras in the park, to obtain a scene semantic segmentation result.

[0079] ​​In some embodiments, in order to improve the scene semantic segmentation accuracy, a test-time augmentation (TTA) technique is adopted to perform horizontal flipping and random rotation transformation on the image frames collected by the data collection module 10 during model inference, and the original image and the transformed image frame are segmented respectively. Then the enhanced image is inversely transformed, and the predicted probability values of each image predicted by the model are averaged to obtain the final scene semantic segmentation result.

[0080] The prohibited violation area (which can also be referred to as a forbidden area) is an area in which a violation behavior is prohibited to appear in the to-be-detected environment or the to-be-detected area. The prohibited violation area includes but is not limited to walls, lawns, roads, public seats, etc. The type of the prohibited violation area is not limited in the present disclosure. The prohibited violation area is related to the to-be-detected environment and the to-be-detected violation behavior. When the to-be-detected environment and the to-be-detected violation behavior are different, the prohibited violation area can also be different.

[0081] In some embodiments, the prohibited violation areas corresponding to different violation behaviors can be partially the same, or can be completely the same, or can be completely different.

[0082] In some embodiments, the user can pre-set a preset forbidden area list. For example, the property management personnel can pre-set the preset forbidden area list to include walls, lawns, and roads.

[0083] In some embodiments, the above step 302 includes performing scene semantic segmentation on the plurality of image frames, and obtaining the prohibited violation areas in the plurality of image frames according to the scene semantic segmentation result and the preset forbidden area list. For example, if the scene semantic segmentation result is in the preset forbidden area list, the prohibited violation areas in the image frames can be obtained.

[0084] In some embodiments, if the plurality of image frames are captured by the same camera at the same angle, the scene semantic segmentation results of the plurality of image frames are the same. Therefore, scene semantic segmentation can be performed on one image in the plurality of images, and the segmentation result can be stored in the data storage module.

[0085] In some embodiments, if the plurality of image frames are captured by the same camera at different angles, the scene semantic segmentation results of the plurality of image frames can be different, and the scene semantic segmentation results of the plurality of images captured at the same angle are the same. Therefore, scene semantic segmentation can be performed on one image in at least one image captured at the same angle.

[0086] In some embodiments, the above step 302 can be performed by the scene semantic segmentation module 11 in the detection system shown in FIG. 1. Figure 1

[0087] ​Step 303: Use a rotating target detection network to detect multiple image frames.

[0088] Rotational object detection networks are used to detect whether an image frame contains an object to be detected.

[0089] For example, step 303 may include using a rotated object detection network to detect multiple image frames and determine whether the multiple image frames contain a target object. The target object may include at least one of the following: pedestrians, clothing, advertisements, etc. This disclosure does not limit the specific type and quantity of the target object; the type of the target object is related to the specific violation to be detected.

[0090] In some embodiments, the rotating object detection network described above can be built based on YOLOv5, which is capable of detecting tilted objects in an image frame. The rotating object detection network adds a rotation angle prediction output for the detection box (also known as the bounding box).

[0091] Figure 4A Here is an architecture diagram of a rotating target detection network 40 according to some embodiments, such as Figure 4A As shown, the rotating target detection network 40 includes an input module 41 and a target detection module 42. The input module 41 is used to input image data from multiple image frames into the rotating target detection network 40. Data transmission is possible between the input module 41 and the target detection module 42.

[0092] The target detection module 42 is used to process image data from multiple image frames to obtain the detection results of the target. For example... Figure 4A As shown, the target detection module 42 includes a backbone network 421, a neck network 422, and a head network 423. The output of the head network 423 includes the predicted rotation angle of the detection box.

[0093] The Backbone network 421 performs extraction operations on image data from multiple image frames to obtain general image features and transmits them to the Neck network 422. Correspondingly, the Neck network 422 receives the general image features sent by the Backbone network 421. It is understood that general image features, in the field of image detection, refer to the image features of general categories of objects obtained by the Backbone network 421 after extracting the original image data during the initial image extraction; this is a well-known concept in the field.

[0094] Neck network 422 extracts image features strongly correlated with the category of the target object from general image features and sends these strongly correlated image features to head network 423. Correspondingly, head network 423 receives the strongly correlated image features sent by neck network 422. It can be understood that strongly correlated image features are image features of objects similar in category to the target object obtained after the general image features have been extracted by neck network 422. The category of the target object can be the detection category set by rotating target detection network 40.

[0095] It should be noted that this disclosure does not limit the specific implementation of the Backbone network 421 for acquiring general image features or the Neck network 422 for acquiring image features strongly correlated with the category of the target object.

[0096] The Head network 423 is used to compute the detection result of the target object based on strongly correlated image features. The detection result of the target object includes, but is not limited to, at least one of the following: the detection box of the target object, the predicted rotation angle of the detection box, the region coordinates of the detection box, or the category of the target object.

[0097] For example, taking the target detection objects as including pedestrians, clothing, and advertisements, such as Figure 4B The output parameters of the Head network 423 shown include three parameters: cls, reg, and ang. cls represents the number of categories of the detected object, with a value of 3. reg represents the coordinates of the detection box region, and includes five parameters: the x and y coordinates of the detection box center point, the length of the long side of the detection box, the length of the short side of the detection box, and the confidence score. Since the confidence score is a parameter corresponding to scene semantic segmentation, used to distinguish whether the detected object is foreground or background, it is not included in the actual calculation. Therefore, in actual calculation, reg can be represented as (x_c, y_c, long, short), where x_c is the x-coordinate of the detection box center point, y_c is the y-coordinate of the detection box center point, long is the length of the long side of the detection box, and short is the length of the short side of the detection box. ang is the angle, with a value of 180, representing 0 to 179 degrees. The angle loss function uses the cross-entropy classification loss.

[0098] like Figure 4B As shown, in Head network 423, 1×1 convolutions ( Figure 4B The Conv1*1 output channel count shown is na*(cls+5+180), where na is the number of anchor boxes, cls is the number of target detection categories, 5 represents the number of parameter categories included in the region coordinates of the reg detection boxes, and 180 is the ang angle value.

[0099] In some embodiments, the detecting the multiple image frames by the rotating object detection network comprises: preprocessing the image frames, extracting feature data of the image frames through a Backbone network and a Neck network, inputting the feature data of the image frames into a Head network to obtain a detection result of the target detection object. The detection result of the target detection object comprises a detection box of the target detection object, region coordinates of the detection box, a predicted rotation angle of the detection box, and a category of the target detection object.

[0100] For example, as shown in Figure 4A , Figure 4B and Figure 5 , when the rotating object detection network is used to detect the image frame as shown in Figure 5 , first, the image frame as shown in Figure 5 is input into an input module 41 of the rotating object detection network for preprocessing. Then, image features of the image frame are extracted through a Backbone network 421 and a Neck network 422. The extracted image features are input into a Head network 423 for prediction to obtain region coordinates of the detection box, a predicted rotation angle θ of the detection box, and a category of the target detection object: advertisement.

[0101] For example, when the Head network 423 calculates the detection result of the target detection object according to the image features, the detection box can be rotated counterclockwise by 0 to 179 degrees along the x-axis, and the angle loss function is used to obtain the detection box that is closest to the target detection object among 180 rotating states, so as to obtain the rotation angle of the detection box and realize the detection of the target detection object in the image frame.

[0102] In some embodiments, as shown in Figure 5 , the predicted rotation angle θ of the detection box can be defined by the long side, and the predicted rotation angle θ of the detection box is the included angle between the long side of the detection box and the x-axis. The predicted rotation angle θ of the detection box can be 0 to 179 degrees.

[0103] It can be understood that, compared with the target detection by using the horizontal bounding box which introduces a large amount of background noise and is difficult to accurately detect the inclined target detection object, the present disclosure can obtain a detection box that is closer to the target detection object by constructing the rotating object detection network to detect the target detection object in the image frame, reduce the redundant information in network learning and strategy making, and accurately detect the inclined target detection object in the image frame.

[0104] In some embodiments, the above step 303 can be performed by a detection and analysis module 12 in the detection system as shown in Figure 1 .

[0105] The disclosure is not limited to the execution order of steps 302 and 303. In some embodiments, step 302 can be executed before step 303, after step 303, or simultaneously with step 303.

[0106] When it is determined in step 303 that at least one of the plurality of image frames includes the target detection object, step 304 can be executed to further determine whether the to-be-detected region has a violation behavior. When it is determined in step 303 that none of the plurality of image frames includes the target detection object, it is indicated that the to-be-detected region has no violation behavior.

[0107] In step 304, when at least one of the plurality of image frames includes the target detection object, the to-be-detected region is determined to have a violation behavior according to the target detection object and the prohibited violation region corresponding to the target detection object.

[0108] In some embodiments, the to-be-detected region can have one or more violation behaviors. The violation behaviors include, but are not limited to, illegal posting of advertisements, illegal airing of clothes, and the like. The target detection objects corresponding to different violation behaviors can be the same or different, and one violation behavior can correspond to one or more target detection objects.

[0109] In some embodiments, since the regions prohibited to appear due to different violation behaviors can be the same or different, different target detection objects can correspond to partially the same, all the same, or all different prohibited violation regions. Embodiments of the disclosure are exemplarily described by taking the case that the prohibited violation regions corresponding to illegal posting of advertisements and illegal airing of clothes are partially the same. The prohibited violation region corresponding to each target detection object can be one type of region or multiple types of regions.

[0110] For example, when the target detection object is an advertisement, the violation region corresponding to the target detection object includes a wall surface and a road. For another example, when the target detection object is clothes, or clothes and pedestrians, the violation region corresponding to the target detection object includes a road and a lawn.

[0111] In some embodiments, if at least one image frame includes at least one target detection object, the to-be-detected region can be further determined to have a violation behavior according to the target detection object and the prohibited violation region corresponding to the target detection object.

[0112] For example, if the at least one image frame includes an advertisement, the system determines whether the target detection region has the behavior of violating the regulation of posting the advertisement according to the advertisement and the forbidden violation region corresponding to the advertisement. For another example, if the at least one image frame includes a pedestrian and clothes, the system determines whether the target detection region has the behavior of violating the regulation of drying the clothes according to the clothes and the forbidden violation region corresponding to the behavior of drying the clothes. For yet another example, if the at least one image frame includes an advertisement, a pedestrian and clothes, the system determines whether the target detection region has the behavior of violating the regulation of posting the advertisement according to the advertisement and the forbidden violation region corresponding to the behavior of posting the advertisement, and determines whether the target detection region has the behavior of violating the regulation of drying the clothes according to the clothes and the forbidden violation region corresponding to the behavior of drying the clothes.

[0113] In some embodiments, the step 304 can be performed by the detection and analysis module 12 in the detection system as shown in FIG. 1. Figure 1

[0114] For example, if the at least one image frame includes a target detection object, the detection and analysis module 12 can determine whether the target detection region has the behavior of violating the regulation according to the intersection area of the detection frame region of the target detection object and the forbidden violation region corresponding to the target detection object.

[0115] In some embodiments, as shown in FIG. 3, the step 304 can include steps 305-307. Figure 6

[0116] The step 305 determines the ratio of the intersection area to the detection frame area of the target detection object according to the intersection area of the detection frame region of the target detection object and the forbidden violation region corresponding to the target detection object.

[0117] For example, if the at least one image frame includes a target detection object, the system can determine whether the target detection object is in the forbidden violation region according to the intersection area of the detection frame region of the target detection object and the forbidden violation region.

[0118] The step 306 determines that the target detection object is in the forbidden violation region when the ratio of the intersection area to the detection frame area of the target detection object is greater than or equal to a first preset threshold.

[0119] In some embodiments, if the ratio of the intersection area to the detection frame area of the target detection object is large (e.g., greater than or equal to the first preset threshold), it can be determined that the target detection object is in the forbidden violation region. If the ratio of the intersection area to the detection frame area of the target detection object is small (e.g., less than the first preset threshold), it can be determined that the target detection object is not in the forbidden violation region.

[0120] ​​For example, the first preset threshold value can be a preset value greater than 0 and less than or equal to 1. For example, when the ratio of the intersection area to the detection frame area of the target detection object is greater than 0.8, that is, 80% of the detection frame of the target detection object is in the prohibited violation area, it is determined that the target detection object is in the prohibited violation area. The specific value of the first preset threshold value is not limited in the embodiments of the present disclosure, and the first preset threshold value is taken as 0.8 for example in the embodiments.

[0121] In some embodiments, in order to ensure that the detection of the violation behavior is more accurate, when it is determined that the target detection object is in the prohibited violation area, whether the violation behavior exists in the to-be-detected area can be further determined according to the detection state count value of the target detection object in the plurality of image frames.

[0122] In step 307, whether the violation behavior exists in the to-be-detected area is determined according to the detection state count value of the target detection object.

[0123] One or more target detection objects can be included in one image frame, and the types of the one or more target detection objects can be the same or different, and can also be partially the same. Each target detection object corresponds to a detection state count value.

[0124] In some embodiments, the initial value of the detection state count value of the target detection object can be a preset value. The initial value of the detection state count value of the target detection object is not limited in the embodiments of the present disclosure, and the initial value of the detection state count value of the target detection object is taken as 0 for example in the following embodiments. It should be noted that when the initial value of the detection state count value of the target detection object is 0, the detection state count value of the target detection object is increased by 1 each time the target detection object appears a violation.

[0125] The detection state count value of the target detection object is used to represent the number of violations of the target detection object, and when the number of violations of the target detection object in the plurality of image frames reaches a certain value (for example, greater than or equal to a third preset threshold value), it can be determined that the violation behavior exists in the to-be-detected area.

[0126] In some embodiments, the step 306 and the step 307 can be combined into one step. The step includes: when the ratio of the intersection area to the detection frame area of the target detection object is greater than or equal to the first preset threshold value, determining whether the violation behavior exists in the to-be-detected area according to the detection state count value of the target detection object. Since when the ratio of the intersection area to the detection frame area of the target detection object is large, it indicates that the target detection object is in the prohibited violation area, therefore, when it is determined that the ratio of the intersection area to the detection frame area of the target detection object is greater than or equal to the first preset threshold value, the detection state count value of the target detection object can be directly used to determine whether the violation behavior exists in the to-be-detected area.

[0127] In some embodiments, when detecting the behavior of illegal posting of advertisements (for example, when the target detection object includes an advertisement), since the posted advertisements are usually static objects in a pasted state, in order to avoid considering garbage, leaves and the like floating in the air as illegal posted advertisements, and to improve the accuracy of detecting illegal posted advertisements, as shown in Figure 7 Step 307 can include steps 308-309.

[0128] Step 308: When the coordinate point displacement of the target detection object in at least one image frame between two adjacent image frames is less than a second preset threshold, the detection state count value of the target detection object is increased by one.

[0129] The coordinate points of the target detection object include five coordinate points of the upper left, lower left, upper right, lower right and center points of the detection frame of the advertisement. The coordinate point displacement of the target detection object refers to the displacement between the coordinate points of the advertisement in the current image frame and the corresponding coordinate points of the advertisement in the previous image frame.

[0130] For example, the coordinate point displacement of the target detection object includes the displacement between the center point of the detection frame of the advertisement in the current image frame and the center point of the detection frame of the advertisement in the previous image frame, the displacement between the upper left point of the detection frame of the advertisement in the current image frame and the upper left point of the detection frame of the advertisement in the previous image frame, and the displacement between the lower left point of the detection frame of the advertisement in the current image frame and the lower left point of the detection frame of the advertisement in the previous image frame. In turn, the coordinate point displacement of the target detection object can include five values.

[0131] In some embodiments, when the coordinate point displacement of the target detection object in at least one image frame between two adjacent image frames is less than a second preset threshold, the detection state count value of the target detection object is increased by one, which can include: when at least one coordinate point of the five coordinate points of the detection frame of the advertisement in the current image frame has a displacement less than the second preset threshold with respect to the corresponding coordinate point of the detection frame of the advertisement in the previous image frame, it can be determined that the target detection object appears once, and the detection state count value of the advertisement is increased by one.

[0132] In some embodiments, the coordinate point displacement of the target detection object in at least one image frame between two adjacent image frames can be calculated according to the detection result of the target detection object obtained in step 303. For example, the coordinate point displacement of the same advertisement in two adjacent image frames can be calculated according to the area coordinates of the advertisement detection frame obtained in step 303.

[0133] It can be understood that, when the target detection object is determined to be in the prohibited violation area, the coordinate point displacement of the target detection object in the at least one image frame and the adjacent two image frames is calculated, and when the displacement is small, it is determined that the target detection object appears once a violation, which can avoid regarding the garbage and leaves floating in the air as the violation posted advertisements, and improve the accuracy of detecting the violation posted advertisements.

[0134] In step 309, when the detection state count value of the target detection object is greater than or equal to the third preset threshold value, it is determined that there is a violation behavior in the to-be-detected area.

[0135] When detecting the violation posted advertisements, it is possible that the posted advertisements fall due to wind, rain and the like. Therefore, if it is determined that the target detection object appears a large number of violations in multiple image frames (for example, the detection state count value of the target detection object is greater than or equal to the third preset threshold value), it is indicated that the target detection object is a firmly pasted advertisement, that is, it is determined that there is a violation posted advertisement behavior in the to-be-detected area.

[0136] In some embodiments, when detecting the violation behavior of airing clothes (for example, when the target detection object includes clothes and pedestrians), the clothes worn by the pedestrians can affect the detection of the violation of airing clothes. In order to further improve the accuracy of detecting the violation of airing clothes, as shown in FIG. 3B, the step 307 can include steps 310-311. Figure 8

[0137] In step 310, when the center point of the detection frame of the clothes in the image frame is outside the detection frame area of at least one pedestrian, the detection state count value of the clothes is increased by one.

[0138] In some embodiments, when the initial value of the detection state count value of the clothes is 0, if the center point of the detection frame of the clothes is outside the detection frame area of at least one pedestrian, it can be determined that the clothes are the violation of airing clothes, rather than the clothes worn by the pedestrians, it is determined that the target detection object appears once a violation, and the detection state count value of the clothes is increased by one. If the center point of the detection frame of the clothes is inside the detection frame area of at least one pedestrian, it can be determined that the target detection object is the clothes worn by the pedestrians, which is not a violation behavior, and the detection state count value of the clothes is not changed.

[0139] ​For example, taking the initial value of the detection state count value of the clothes as 0, if the center point of the detection frame of the clothes is outside the detection frame area of at least one pedestrian in the first image frame, the detection state count value of the clothes is increased by 1. If the center point of the detection frame of the clothes is inside the detection frame area of at least one pedestrian in the second image frame, the detection state count value of the clothes is not changed. If the center point of the detection frame of the clothes is outside the detection frame area of at least one pedestrian in the third image frame, the detection state count value of the clothes is increased by 1 again. In this way, the detection state count value of the clothes in multiple image frames can be obtained. The detection state count value of the clothes represents the number of violations of the clothes.

[0140] For example, in 10 image frames, when the center point of the detection frame of the clothes is outside the detection frame area of the pedestrian, it is determined that the clothes is a target detection object in violation, and the detection state count value indicating the number of violations of the clothes is increased by 1. When the clothes is inside the detection frame area of the pedestrian, it is determined that the clothes is clothes worn by the pedestrian, and the detection state count value of the clothes is kept unchanged. In this way, if the center point of the detection frame of the clothes is outside the detection frame area of the pedestrian in 8 of the 10 image frames, the detection state count value of the clothes can be obtained as 8.

[0141] In some embodiments, the area of the detection frame of the at least one pedestrian is greater than or equal to the fifth preset threshold, and the center point of the detection frame of the at least one pedestrian is inside the prohibited violation area corresponding to the target detection object. It can be understood that, in order to improve the speed and accuracy of detecting the clothes in violation, the detection frame of the pedestrian with a smaller area in the image frame and the detection frame of the pedestrian with a center point not inside the prohibited violation area can be deleted, and the detection frame of the pedestrian with a larger area in the image frame and the detection frame of the pedestrian with a center point inside the prohibited violation area (i.e., the detection frame of the at least one pedestrian) can be retained. Then, it is determined whether the center point of the detection frame of the clothes is inside the detection frame area of the retained at least one pedestrian, to determine whether the clothes is clothes worn by the pedestrian or clothes in violation.

[0142] In some embodiments, the center point of the detection frame of the clothes in the image frame can be calculated according to the detection result of the target detection object obtained in step 303. For example, the center point of the detection frame of the clothes can be calculated according to the area coordinates of the detection frame of the clothes and the area coordinates of the detection frame of the pedestrian obtained in step 303.

[0143] In step 311, in a case where the detection state count value of the clothes is greater than or equal to the fourth preset threshold, it is determined that there is a violation behavior in the detection area.

[0144] Since the clothes being detected for the violation of the drying rule may coincide with the passerby, in order to improve the accuracy of the detection of the clothes being dried in violation of the drying rule, the clothes being dried in violation of the drying rule can be determined through multiple images. If it is determined that the target detection object appears in violation in the multiple images for a large number of times (for example, the detection state count value of the clothes is greater than or equal to a fourth preset threshold value), it is indicated that the clothes are the clothes being dried in violation of the drying rule, rather than the clothes worn by the passerby, and it can be determined that the clothes being dried in violation of the drying rule exist in the to-be-detected region.

[0145] In some embodiments, as shown in FIG. 3, in addition to the steps 310-311 described above, the step 307 can further include a step 312, which can be performed before the step 310 described above. Figure 9

[0146] The step 312 deletes the detection frame of the passerby whose area is less than a fifth preset threshold value or whose center point is outside the prohibited violation region.

[0147] The area of the detection frame of the passerby being less than the fifth preset threshold value or the center point of the detection frame of the passerby being outside the prohibited violation region includes that the body of the passerby in the image frame is incomplete, or the passerby is far away from the camera, or the passerby is not in the prohibited violation region, and the like. When the area of the detection frame of the passerby is less than the fifth preset threshold value or the center point of the detection frame of the passerby is outside the prohibited violation region, the clothes worn by the passerby will not affect the detection of the clothes being dried in violation of the drying rule, and thus the detection frame of the passerby can be deleted. The passerby satisfying at least one of the above two conditions is irrelevant to the detection of the clothes being dried in violation of the drying rule, and thus is deleted.

[0148] In order to improve the speed and accuracy of the detection of the clothes being dried in violation of the drying rule, the detection frame of the passerby in the image frame irrelevant to the detection of the clothes being dried in violation of the drying rule (for example, the detection frame of the passerby with a small area in the image frame, and the detection frame of the passerby with a center point outside the prohibited violation region) can be deleted, and the detection frame of the passerby relevant to the detection of the clothes being dried in violation of the drying rule (for example, the detection frame of the passerby with a large area in the image frame, and the detection frame of the passerby with a center point in the prohibited violation region) can be retained. The retained detection frame of the passerby is the detection frame of at least one passerby in the step 310 described above. Then, whether the center point of the detection frame of the clothes is in the area of the retained detection frame of the at least one passerby is judged through the step 310, to determine whether the clothes are the clothes worn by the passerby or the clothes being dried in violation of the drying rule.

[0149] In some embodiments, when it is determined that the violation behavior exists in the to-be-detected region, as shown in FIG. 3, in addition to the steps 301-304 described above, the method can further include steps 313-315. Figure 10

[0150] The step 313 determines whether the alarm identifier indicates that no alarm occurs.​​

[0151] The alarm identifier is used to indicate whether an alarm has occurred. The alarm identifier can be represented by a Boolean value or a numerical value.

[0152] For example, when the alarm identifier is represented by a Boolean value, true indicates that an alarm has occurred, and false indicates that an alarm has not occurred. For another example, when the alarm identifier is represented by a numerical value, 1 indicates that an alarm has occurred, and 0 indicates that an alarm has not occurred. The specific value of the alarm identifier indicating whether an alarm has occurred is not limited in the present disclosure.

[0153] Step 314: When the alarm identifier indicates that no alarm has occurred, report the alarm information and change the alarm identifier.

[0154] For example, when the alarm identifier is false, indicating that no alarm has occurred, report the alarm information and change the alarm identifier to true. For another example, when the alarm identifier is 0, indicating that no alarm has occurred, report the alarm information and change the alarm identifier to 1.

[0155] Step 315: When the alarm identifier indicates that an alarm has occurred, and the time since the last alarm is greater than a preset time, report the alarm information.

[0156] For example, when the alarm identifier is true or the alarm identifier is 1, indicating that an alarm has occurred, calculate the time difference between the current time and the last alarm time. If the time difference between the current time and the last alarm time is greater than a preset time, report the alarm information.

[0157] In some embodiments, the action of reporting the alarm information in steps 314 and 315 described above can be performed by the sending module 14 in the detection system shown in Figure 1 .

[0158] In some embodiments, the detection system can also analyze the areas where frequent alarms occur within a period of time (for example, one month) according to the alarm records, and send a ‘warning sign setting’ prompt to the property management center.

[0159] Some embodiments of the present disclosure provide a method for detecting illegal behavior, which is used to detect whether an advertisement is illegally posted, such as Figure 11 as shown in the figure, the detection method comprises steps 1 to 11.

[0160] Step 1: Obtain an image frame of a to-be-detected area.

[0161] Step 2: Perform scene semantic segmentation on the image frame to obtain a prohibited illegal area in the image frame.

[0162] When the camera angle is unchanged, the multiple image frames captured by the camera can be subjected to one scene semantic segmentation. When the camera angle changes, the image frames of the to-be-detected region captured by the camera can be subjected to scene semantic segmentation again.

[0163] Step 3, detecting a target detection object in the image frame: an advertisement.

[0164] Step 4, judging whether a ratio of the intersection area to the area of the detection frame of the advertisement is greater than or equal to a first preset threshold value.

[0165] The intersection area is an area of intersection between the detection frame of the advertisement and the forbidden area corresponding to the advertisement.

[0166] If the ratio of the intersection area to the area of the detection frame of the advertisement is greater than or equal to the first preset threshold value, step 5 is performed. If the ratio of the intersection area to the area of the detection frame of the advertisement is less than the first preset threshold value, step 3 is repeatedly performed.

[0167] Step 5, judging whether a coordinate point displacement of the advertisement in the adjacent two image frames is less than a second preset threshold value.

[0168] If the coordinate point displacement of the advertisement in the adjacent two image frames is less than the second preset threshold value, step 6 is performed. If the coordinate point displacement of the advertisement in the adjacent two image frames is greater than or equal to the second preset threshold value, step 3 is repeatedly performed.

[0169] Step 6, increasing a detection state count value of the advertisement by one.

[0170] Step 7, judging whether the detection state count value of the advertisement is greater than or equal to a third preset threshold value.

[0171] If the detection state count value of the advertisement is greater than or equal to the third preset threshold value, step 8 is performed. If the detection state count value of the advertisement is less than the third preset threshold value, step 3 is repeatedly performed.

[0172] Step 8, judging whether the alarm identifier indicates that no alarm has occurred.

[0173] If the alarm identifier indicates that no alarm has occurred, steps 9-10 are performed. If the alarm identifier indicates that an alarm has occurred, step 11 is performed.

[0174] Step 9, changing the alarm identifier to indicate that an alarm has occurred.

[0175] Step 10, reporting the alarm information.

[0176] In some embodiments, when the alarm identifier is changed to indicate that an alarm has occurred, the alarm information can be reported after the first detection information is reported.

[0177] Step 11: judging whether the difference between the current time and the last alarm time is greater than the preset time.

[0178] If the difference between the current time and the last alarm time is greater than the preset time, step 10 is continued to be executed.

[0179] Some embodiments of the present disclosure provide another method for detecting whether a violation behavior occurs, such as whether a clothes airing violation occurs. Figure 12 As shown in the method, the method comprises steps a to n.

[0180] Step a: obtaining an image frame of a to-be-detected region.

[0181] Step b: performing scene semantic segmentation on the image frame to obtain a forbidden violation region in the image frame.

[0182] When the camera angle is unchanged, a plurality of image frames captured by the camera can continue to perform scene semantic segmentation once. When the camera angle changes, scene semantic segmentation can be performed again on the image frames of the to-be-detected region captured by the camera.

[0183] Step c: detecting a target detection object, i.e., a pedestrian, in the image frame.

[0184] Step d: deleting a pedestrian detection frame whose area is less than a fourth preset threshold.

[0185] Step e: deleting a pedestrian detection frame whose center point is outside the forbidden region, and step h is continued to be executed.

[0186] Step f: detecting a target detection object, i.e., clothes, in the image frame.

[0187] Step g: judging whether a ratio of an intersection area to an area of a clothes detection frame is greater than or equal to a first preset threshold.

[0188] The intersection area is an area of intersection between the clothes detection frame and a forbidden region corresponding to the clothes.

[0189] If the ratio of the intersection area to the area of the clothes detection frame is greater than or equal to the first preset threshold, step h is continued to be executed. If the ratio of the intersection area to the area of the clothes detection frame is less than the first preset threshold, step f is repeatedly executed.

[0190] Step h: judging whether a center point of the clothes detection frame is outside at least one pedestrian detection frame.

[0191] The area of the at least one pedestrian detection frame is greater than or equal to a fifth preset threshold, and a center point of the at least one pedestrian detection frame is inside a forbidden violation region corresponding to the target detection object.

[0192] If the clothing detection box center point is outside any pedestrian detection box, step i is continued to be executed. If the clothing detection box center point is inside all pedestrian detection boxes, step f is repeatedly executed.

[0193] Step i: the detection state count value of the clothing is increased by one.

[0194] Step j: whether the detection state count value of the clothing is greater than or equal to a fourth preset threshold value is judged.

[0195] If the detection state count value of the clothing is greater than or equal to the fourth preset threshold value, step k is continued to be executed. If the detection state count value of the clothing is less than the fourth preset threshold value, step f is repeatedly executed.

[0196] Step k: whether the alarm identifier indicates that no alarm has occurred is judged.

[0197] If the alarm identifier indicates that no alarm has occurred, steps l and m are continued to be executed. If the alarm identifier indicates that an alarm has occurred, step n is continued to be executed.

[0198] Step l: the alarm identifier is changed to indicate that an alarm has occurred.

[0199] In some embodiments, when the alarm identifier is changed to indicate that an alarm has occurred, the alarm information can be reported after the first detection information is reported.

[0200] Step m: the alarm information is reported.

[0201] In some embodiments, when the alarm identifier is changed to indicate that an alarm has occurred, the alarm information can be reported after the first detection information is reported.

[0202] Step n: whether the difference between the current time and the last alarm time is greater than a preset time is judged.

[0203] If the difference between the current time and the last alarm time is greater than the preset time, step m is continued to be executed.

[0204] It should be noted that the present disclosure does not limit the numerical values of the above-mentioned first preset threshold value, second preset threshold value, third preset threshold value, fourth preset threshold value, fifth preset threshold value, and preset time. For the same preset threshold value (for example, the first preset threshold value), one numerical value can be set when detecting different violations, or multiple numerical values can be set. For example, when detecting the violation of posting advertisements, the first preset threshold value can be set to 0.9, and when detecting the violation of airing clothes, the first preset threshold value can be set to 0.8. That is, the first preset threshold value can include multiple numerical values when detecting different violations.

[0205] Some embodiments of the present disclosure provide a violation detection device, such as Figure 13As shown, the violation detection apparatus 1300 includes a processor 1301 and a transceiver 1302.

[0206] The transceiver 1302 is configured to acquire a plurality of image frames corresponding to a to-be-detected region.

[0207] The processor 1301 is configured to perform scene semantic segmentation on the plurality of image frames to obtain a prohibited violation region in the plurality of image frames, detect the plurality of image frames by using a rotated object detection network, and determine that the to-be-detected region has a violation behavior in a case where at least one image frame in the plurality of image frames includes a target detection object, according to the target detection object and a prohibited violation region corresponding to the target detection object.

[0208] The target detection object includes at least one of an advertisement, a piece of clothing, or a pedestrian.

[0209] The processor 1301 is further configured to determine a ratio of an intersection area of a detection frame region of the target detection object and the prohibited violation region corresponding to the target detection object, determine that the target detection object is in the prohibited violation region in a case where the ratio of the intersection area and a detection frame area of the target detection object is greater than or equal to a first preset threshold, and determine that the to-be-detected region has the violation behavior according to a detection state count value of the target detection object.

[0210] The processor 1301 is further configured to increase the detection state count value of the target detection object by one in a case where a coordinate point of the target detection object in adjacent two image frames in the at least one image frame is displaced by less than a second preset threshold, and determine that the to-be-detected region has the violation behavior in a case where the detection state count value of the target detection object is greater than or equal to a third preset threshold.

[0211] The processor 1301 is further configured to increase the detection state count value of the piece of clothing by one in a case where a detection frame center point of the piece of clothing is outside a detection frame region of at least one pedestrian, and determine that the to-be-detected region has the violation behavior in a case where the detection state count value of the piece of clothing is greater than or equal to a fourth preset threshold.

[0212] An area of the detection frame of the at least one pedestrian is greater than or equal to a fifth preset threshold, and a center point of the detection frame of the at least one pedestrian is in the prohibited violation region corresponding to the target detection object.

[0213] The processor 1301 is further configured to delete a detection frame in the detection frame of the pedestrian, which has an area less than the fifth preset threshold or a center point outside the prohibited violation region.

[0214] In some embodiments, the rotated object detection network described above includes a Backbone network, a Neck network, and a Head network, and an output of the Head network includes a predicted rotation angle of a detection frame.

[0215] The processor 1301 is configured to: pre-process the image frame, extract feature data of the image frame through the Backbone network and the Neck network, input the feature data of the image frame into the Head network, and obtain a detection result of a target detection object. The detection result of the target detection object includes at least one of a detection frame of the target detection object, a region coordinate of the detection frame, a predicted rotation angle of the detection frame, or a category of the target detection object.

[0216] The transceiver 1302 is further configured to: when the alarm identifier indicates that no alarm has occurred, send the alarm information and change the alarm identifier; and when the alarm identifier indicates that the alarm has occurred and the time since the last alarm is greater than a preset time, send the alarm information.

[0217] The transceiver 1302 can be a communication interface, for example, an input and output interface. For example, when the violation behavior detection apparatus 1300 is a chip, the transceiver 1302 is a communication interface. The transceiver 1302 can also be a radio frequency unit. For example, when the violation behavior detection apparatus 1300 is a terminal device, the transceiver 1302 can be a radio frequency unit connected with an antenna.

[0218] In some embodiments, the above-described violation behavior detection apparatus 1300 can further include a memory 1303 configured to store program codes and data corresponding to any of the above-described violation behavior detection methods. The memory 1303 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), and the like.

[0219] In some embodiments, the above-described violation behavior detection apparatus 1300 can be Figure 2 The violation behavior detection apparatus 20 shown in FIG. 1, the above-described Figure 2 The descriptions of all related contents of the components involved can be cited to the descriptions of the corresponding components in the above-described violation behavior detection apparatus 1300, and will not be repeated here. Figure 13 The descriptions of all related contents of the components involved can be cited to the descriptions of the corresponding components in the above-described violation behavior detection apparatus 1300, and will not be repeated here.

[0220] Some embodiments of the present disclosure provide a computer readable storage medium (for example, a non-transitory computer readable storage medium) having computer program instructions stored therein, which, when executed on a computer (for example, a violation behavior detection apparatus), cause the computer to perform the violation behavior detection method as described in any of the above-described embodiments.

[0221] Exemplarily, the computer readable storage medium described above can include, but is not limited to, a magnetic storage device (for example, a hard disk, a floppy disk or a magnetic tape, etc.), an optical disk (for example, a CD (Compact Disk), a DVD (Digital Versatile Disk), etc.), a smart card and a flash memory device (for example, an EPROM (Erasable Programmable Read-Only Memory), a card, a stick or a key drive, etc.). The various computer readable storage media described in the present disclosure can represent one or more devices and / or other machine readable storage media for storing information. The term "machine readable storage medium" can include, but is not limited to, a wireless channel and various other media capable of storing, containing and / or carrying instructions and / or data.

[0222] Some embodiments of the present disclosure also provide a computer program product, for example, the computer program product is stored on a non-transitory computer readable storage medium. The computer program product includes computer program instructions, when the computer program instructions are executed on a computer (for example, a violation behavior detection device), the computer program instructions cause the computer to perform the violation behavior detection method described in the above embodiments.

[0223] Some embodiments of the present disclosure also provide a computer program. When the computer program is executed on a computer, the computer program causes the computer to perform the violation behavior detection method described in the above embodiments.

[0224] The beneficial effects of the computer readable storage medium, the computer program product and the computer program described above are the same as the beneficial effects of the violation behavior detection method described in some embodiments described above, and will not be repeated here.

[0225] The above is merely specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto, any person skilled in the art can think of changes or replacements within the technical range disclosed in the present disclosure, which should be covered in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for detecting a violation behavior, comprising: obtaining a plurality of image frames corresponding to a region to be detected; performing scene semantic segmentation on the plurality of image frames to obtain a prohibited violation region in the plurality of image frames; detecting the plurality of image frames by using a rotating object detection network, the rotating object detection network being configured to detect whether the image frames include a target detection object; in a case where at least one image frame in the plurality of image frames includes the target detection object, determining that the region to be detected has a violation behavior according to the target detection object and the prohibited violation region corresponding to the target detection object; the determining that the region to be detected has a violation behavior according to the target detection object and the prohibited violation region corresponding to the target detection object comprises: determining a ratio of an intersection area of a bounding box region of the target detection object and the prohibited violation region corresponding to the target detection object; in a case where the ratio of the intersection area and the bounding box area of the target detection object is greater than or equal to a first preset threshold, determining that the target detection object is in the prohibited violation region; determining that the region to be detected has a violation behavior according to a detection state count value of the target detection object; in a case where the target detection object includes a piece of clothing and a pedestrian, the determining that the region to be detected has a violation behavior according to the detection state count value of the target detection object comprises: in a case where a center point of the bounding box of the piece of clothing is outside a bounding box region of the at least one pedestrian, increasing the detection state count value of the piece of clothing by one; in a case where the detection state count value of the piece of clothing is greater than or equal to a fourth preset threshold, determining that the region to be detected has a violation behavior.

2. The method of claim 1, wherein, an area of the bounding box of the at least one pedestrian is greater than or equal to a fifth preset threshold, and a center point of the bounding box of the at least one pedestrian is in the prohibited violation region corresponding to the target detection object.

3. The method of claim 1 or 2, further comprising: deleting a bounding box in the bounding box of the pedestrian that has an area less than a fifth preset threshold or a center point that is outside the prohibited violation region.

4. The method of claim 1 or 2, wherein, the rotating object detection network comprises a Backbone network, a Neck network and a Head network, and an output of the Head network comprises a predicted rotation angle of a bounding box.

5. The method of claim 4, wherein, the detecting the plurality of image frames by using the rotating object detection network comprises: performing preprocessing on the image frames, extracting feature data of the image frames through the Backbone network and the Neck network; inputting the feature data of the image frames into the Head network to obtain a detection result of the target detection object, the detection result of the target detection object comprising at least one of a bounding box of the target detection object, a region coordinate of the bounding box, a predicted rotation angle of the bounding box, or a category of the target detection object.

6. The method of claim 1 or 2, further comprising: in a case where an alarm indication indicates that no alarm has occurred, reporting alarm information and changing the alarm indication. When the alarm indication indicates that an alarm has occurred, and the time since the last alarm is greater than a preset time, the alarm information is reported.

7. A device for detecting a violation, comprising: A processor and a transceiver; The transceiver is configured to obtain a plurality of image frames corresponding to a to-be-detected region; The processor is configured to: perform scene semantic segmentation on the plurality of image frames to obtain a prohibited violation region in the plurality of image frames; detect the plurality of image frames using a rotating target detection network; in a case where at least one of the plurality of image frames includes a target detection object, determine that the to-be-detected region has a violation behavior according to the target detection object and the prohibited violation region corresponding to the target detection object; The processor is configured to: determine a ratio of an intersection area of the bounding box region of the target detection object and the prohibited violation region corresponding to the target detection object; when the ratio of the intersection area and the bounding box area of the target detection object is greater than or equal to a first preset threshold, determine that the target detection object is in the prohibited violation region; determine that the to-be-detected region has a violation behavior according to a detection state count value of the target detection object; The processor is specifically configured to: when the detection frame center point of the clothes is outside the detection frame region of at least one pedestrian, increase the detection state count value of the clothes by one; in a case where the detection state count value of the clothes is greater than or equal to a fourth preset threshold, determine that there is a violation behavior in the to-be-detected region.

8. The apparatus of claim 7, wherein, The area of the detection frame of the at least one pedestrian is greater than or equal to a fifth preset threshold, and the center point of the detection frame of the at least one pedestrian is in the prohibited violation region corresponding to the target detection object.

9. The apparatus of claim 7 or 8, wherein the processor is further configured to: delete the detection frame in the detection frame of the pedestrian that has an area less than a fifth preset threshold or a center point outside the prohibited violation region.

10. The apparatus of claim 7 or 8, wherein, The rotating target detection network includes a Backbone network, a Neck network, and a Head network, and the output of the Head network includes a predicted rotation angle of a detection frame.

11. The apparatus of claim 10, wherein, The processor is further configured to: perform preprocessing on the image frame, extract feature data of the image frame through the Backbone network and the Neck network; input the feature data of the image frame into the Head network to obtain a detection result of a target detection object; the detection result of the target detection object includes at least one of a detection frame of the target detection object, a region coordinate of the detection frame, a predicted rotation angle of the detection frame, or a category of the target detection object.

12. A non-transitory computer-readable storage medium storing computer program instructions, the computer program instructions being executed by a detection device to implement the violation behavior detection method of any one of claims 1-6.

Images