Method and device for training violation detection model, method and device for violation detection
By training a violation detection model using a small number of labeled samples and a large number of unlabeled samples combined with perturbation information, the problems of high model development cost and poor generalization in existing technologies are solved, and effective violation detection is achieved in different scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CORP LTD
- Filing Date
- 2022-11-22
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, violation detection models based on object detection or instance segmentation require a large number of labeled samples for training, which is costly and lacks generalization and transferability, and cannot meet the needs of different violation scenarios.
The violation detection model is trained using a small number of labeled samples and a large number of unlabeled samples combined with perturbation information. Through an encoder, feature difference module and decoder, the perturbation information is used to reduce the output difference between the detection model under perturbation and perturbation conditions. Cross-entropy and mean squared error loss functions are used for training optimization.
It reduces model development costs, improves model generalization and transferability, and enables effective violation detection in different scenarios.
Smart Images
Figure CN115731615B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and in particular to training methods and apparatus for violation detection models, violation detection methods and apparatus, electronic devices, and computer-storable media. Background Technology
[0002] One of the main goals of smart city management is to detect various violations in the city's operation, such as illegal occupation of fire lanes, unauthorized pets, pedestrians jaywalking, and illegal posting of advertisements. As cities grow larger, relying solely on manual patrols to identify such violations requires a significant investment of human resources. However, fixed video surveillance equipment has now penetrated every corner of the city. By automatically detecting violations and anomalies through the collected image information, labor costs can be greatly reduced, intervention measures can be taken more quickly, and the efficiency of smart city management can be improved.
[0003] In related technologies, image processing algorithms such as object detection algorithms or instance segmentation algorithms are used to detect violations. Summary of the Invention
[0004] In related technologies, object detection algorithms use a bounding box to enclose abnormal targets. However, since the bounding box does not directly fit the outline of the violation target, using only the bounding box may lead to false positives, and it cannot meet the requirement of accurately determining the region of the abnormal target in some scenarios. Instance segmentation algorithms can output the outline of abnormal targets, but the annotation is complex. Regardless of whether the algorithm is based on object detection or instance segmentation, a large number of labeled samples are needed to train a usable violation detection model. Moreover, different violation scenarios require different object detection models, increasing the cost of model development and lacking good generalization and transferability.
[0005] To address the aforementioned technical issues, this disclosure proposes a solution that can reduce the cost of model development and improve the generalizability and transferability of models.
[0006] According to a first aspect of this disclosure, a method for training a violation detection model is provided, comprising: acquiring training image samples, wherein the training image samples include a first sample size of labeled samples and a second sample size of unlabeled samples, wherein both the labeled samples and the unlabeled samples include a baseline image without violation events and a violation image with violation events, the labeled samples further including a label image describing the violation events, the first sample size being smaller than the second sample size, and the baseline image and the violation image being used to describe regions at the same location; performing a first training on a violation detection model for detecting violation events based on the labeled samples; and performing a second training on the violation detection model obtained from the first training based on the labeled samples and the unlabeled samples, wherein, in the second training, at least one perturbation information is added, the perturbation information being used to reduce the difference between the output of the violation event detection model under the unperturbed condition and the output of the violation event detection model under the perturbed condition in the second training.
[0007] In some embodiments, performing a second training on the violation detection model obtained from the first training includes: using the violation detection model to detect violation events in the violation images based on the benchmark image and the violation image in the training image samples, and obtaining a first output image; for the unlabeled samples, using the violation detection model to detect violation events in the violation images while adding the at least one perturbation information based on the corresponding benchmark image and the violation image, and obtaining a second output image; for the labeled samples, determining the loss value of a first loss function based on the corresponding first output image and the label image, wherein the loss value characterizes the degree of difference between the corresponding first output image and the label image; for the unlabeled samples, determining the loss value of a second loss function based on the corresponding first output image and the second output image, wherein the loss value characterizes the degree of difference between the corresponding first output image and the second output image; and performing a second training on the violation detection model obtained from the first training based on the loss value of the first loss function and the loss value of the second loss function.
[0008] In some embodiments, the violation detection model includes an encoder, a feature difference module, and a decoder. Based on a corresponding reference image and a violation image, while adding at least one perturbation information, the violation detection model detects violation events in the violation image to obtain a second output image. This includes: using the encoder to determine a first feature corresponding to the reference image and a second feature corresponding to the violation image, based on the reference image and the violation image in the training image samples; using the feature difference module to determine difference features based on the first and second features of the training image samples; adding at least one perturbation information to the corresponding difference features for the unlabeled samples to obtain perturbed difference features; and using the decoder to detect violation events in the violation image based on the perturbed difference features corresponding to the unlabeled samples to obtain the second output image.
[0009] In some embodiments, adding the at least one perturbation information to the corresponding differential features includes at least one of the following: adding random noise to the corresponding differential features, wherein the random noise has the same dimension as the differential features; randomly removing features in the corresponding differential features whose normalized feature values are less than a pre-configured feature threshold; using a random erasure algorithm to erase features in the corresponding differential features corresponding to the salient regions of the first output image corresponding to the unlabeled samples; performing mask operations on the corresponding differential features based on the salient regions of the first output image corresponding to the unlabeled samples; and adding virtual adversarial perturbations to the corresponding differential features.
[0010] In some embodiments, detecting violations in the violation images and obtaining a first output image using the violation detection model based on the baseline image and the violation image in the training image samples includes: detecting violations in the violation images using a decoder based on the differential features corresponding to the training image samples, and obtaining a first output image.
[0011] In some embodiments, performing a second training on the violation detection model obtained from the first training based on the loss value of the first loss function and the loss value of the second loss function includes: weighting the loss value of the second loss function according to a regularization parameter related to the number of iterations, wherein the regularization parameter is positively correlated with the number of iterations; determining a total loss value based on the sum of the loss value of the first loss function and the weighted loss value of the second loss function, wherein, when multiple perturbation information is added, there are multiple loss values of the second loss function; and performing a second training on the violation detection model obtained from the first training based on the total loss value.
[0012] In some embodiments, the first loss function is the cross-entropy loss function, and the second loss function is the mean squared error loss function.
[0013] In some embodiments, the first training of a violation detection model for detecting violations based on the labeled samples includes: using the violation detection model to detect violations in the violation images based on the baseline image and the violation image in the labeled samples, and obtaining a first output image; for the labeled samples, determining the loss value of a first loss function based on the corresponding first output image and the label image, wherein the loss value characterizes the degree of difference between the corresponding first output image and the label image; and performing a first training on the violation detection model obtained from the first training based on the loss value of the first loss function.
[0014] In some embodiments, the labeled samples and the unlabeled samples further include regions of interest that match the violation scenario. These regions of interest are part of the location region and are the core region where the violation event occurs in the violation scenario.
[0015] In some embodiments, the ratio of the first sample size to the second sample size is less than or equal to 1 / 9.
[0016] According to a second aspect of this disclosure, a violation detection method is provided, comprising: acquiring a target scene image to be detected; using a violation detection model to detect violation events in the image to be detected, and obtaining an output image, wherein the violation detection model is obtained by the training method described in any of the above embodiments.
[0017] In some embodiments, there are multiple images to be detected, and the violation detection method further includes: accumulating violation events based on the time difference between the multiple images to be detected and the output image; and determining that there are violation events in the target scene and issuing an alarm when the accumulated violation events exceed a preset time threshold.
[0018] According to a third aspect of this disclosure, a training apparatus for a violation detection model is provided, comprising: an acquisition module configured to acquire training image samples, wherein the training image samples include a first sample size of labeled samples and a second sample size of unlabeled samples, wherein both the labeled samples and the unlabeled samples include a baseline image without violation events and a violation image with violation events, the labeled samples further including a label image describing the violation events, the first sample size being smaller than the second sample size, and the baseline image and the violation image being used to describe regions at the same location; a first training module configured to perform a first training on a violation detection model for detecting violation events based on the labeled samples; and a second training module configured to perform a second training on the violation detection model obtained from the first training based on the labeled samples and the unlabeled samples, wherein, in the second training, at least one perturbation information is added, the perturbation information being used to reduce the difference between the output of the violation event detection model under the unperturbed condition and the output of the violation event detection model under the perturbed condition in the second training.
[0019] According to a fourth aspect of this disclosure, a violation detection apparatus is provided, comprising: an acquisition module configured to acquire an image to be detected; and a detection module configured to detect violation events in the image to be detected using a violation detection model to obtain an output image, wherein the violation detection model is obtained by the training method described in any of the above embodiments.
[0020] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the training method or violation detection method described in any of the above embodiments based on instructions stored in the memory.
[0021] According to a sixth aspect of this disclosure, a computer-storeable medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the training method or violation detection method described in any of the above embodiments.
[0022] The above embodiments can reduce the cost of model development and improve the generalizability and transferability of the model. Attached Figure Description
[0023] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.
[0024] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:
[0025] Figure 1 This is a flowchart illustrating a training method for a violation detection model according to some embodiments of the present disclosure;
[0026] Figure 2 This is a schematic diagram illustrating labeled samples according to some embodiments of the present disclosure;
[0027] Figure 3 This is a schematic diagram illustrating a first training process according to some embodiments of the present disclosure;
[0028] Figure 4 This is a partial schematic diagram illustrating a second training process according to some embodiments of the present disclosure;
[0029] Figure 5 This is a flowchart illustrating a violation detection method according to some embodiments of the present disclosure;
[0030] Figure 6 This is a block diagram illustrating a training apparatus for a violation detection model according to some embodiments of the present disclosure;
[0031] Figure 7 This is a block diagram illustrating a violation detection apparatus according to some embodiments of the present disclosure;
[0032] Figure 8 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure;
[0033] Figure 9 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. Detailed Implementation
[0034] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0035] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0036] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0037] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0038] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0040] Figure 1 This is a flowchart illustrating a method for training a violation detection model according to some embodiments of the present disclosure.
[0041] like Figure 1 As shown, the training method for the violation detection model includes steps S110-S130.
[0042] In step S110, training image samples are acquired. The training image samples include a first number of labeled samples and a second number of unlabeled samples. Both the labeled and unlabeled samples include a baseline image without violations and a violation image with violations. The labeled samples also include tag images describing the violations. The baseline image and the violation image are used to describe the same location area. For example, the baseline image and the violation image are both monitoring images of the same location area from the same monitoring device at different times.
[0043] Figure 2 This is a schematic diagram illustrating labeled samples according to some embodiments of the present disclosure.
[0044] like Figure 2 As shown, the labeled samples include a baseline image describing the location of the fire lane, in which no violations occur. The labeled samples also include images describing violations in the area where the fire lane is located, showing vehicles illegally blocking the fire lane. The labeled samples also include tagged images describing the violations. Figure 2 As shown, the label image describes the violation by identifying the outline of the violating target.
[0045] The first sample size is smaller than the second sample size. In some embodiments, the ratio of the first sample size to the second sample size is less than or equal to 1 / 9. In this case, the labeled samples are a small number of labeled samples compared to the unlabeled samples. For example, the number of labeled samples is approximately 200 to 300.
[0046] In some embodiments, the labeled samples and the unlabeled samples further include regions of interest (ROIs) matching the violation scenario. These ROIs are part of the location region and represent the core region where the violation event occurs within the violation scenario. The ROIs can be customized based on the violation scenario. For example, refer to... Figure 2 The areas of interest for violations include fire lanes marked on the road surface.
[0047] For example, a small number of labeled samples are represented as D. lA large number of unlabeled samples are represented as D. ul The i-th group of labeled sample pairs contains the baseline image. illegal images Areas of interest that violate regulations and label images
[0048] return Figure 1 In step S120, the violation detection model used to detect violation events is trained for the first time based on the labeled samples.
[0049] In some embodiments, the first training of a violation detection model for detecting violations based on the labeled samples includes the following steps.
[0050] First, based on the baseline image and the violation image in the labeled sample, the violation detection model is used to detect violation events in the violation image to obtain the first output image.
[0051] In some embodiments, the violation detection model includes an encoder, a feature difference module, and a decoder. The output of the encoder serves as the input to the feature difference module, and the output of the feature difference module serves as the input to the decoder. In this case, firstly, based on the baseline image and the violation image in the labeled samples, the encoder determines a first feature corresponding to the baseline image and a second feature corresponding to the violation image, respectively. Then, based on the first and second features of the training image samples, the feature difference module determines the difference features. Finally, based on the difference features corresponding to the labeled samples, the decoder detects the violation event in the violation image, obtaining a first output image. In some embodiments, the encoder is a ResNet50.
[0052] Then, for the labeled sample, the loss value of the first loss function is determined based on the corresponding first output image and label image. This loss value characterizes the degree of difference between the corresponding first output image and label image.
[0053] Finally, based on the loss value of the first loss function, the violation detection model obtained from the first training is subjected to the first training.
[0054] In some embodiments, the first training is preliminary model training. During preliminary model training, labeled samples D are used. l Perform model training.
[0055] Figure 3 This is a schematic diagram illustrating a first training process according to some embodiments of the present disclosure.
[0056] like Figure 3 As shown, surveillance images from different times are selected. (The baseline image set in the labeled samples) and (The set of non-compliant images in the labeled samples) are used as inputs to the encoder of the model to obtain the encoded features. and Will and As input to the feature difference module, the difference features are obtained. The feature difference module implements feature... and The decoder compares the differential features. Decode the data to obtain the predicted violations within the region of interest. (Output image). During the first training process, the cross-entropy loss function can be used to measure the difference between the model output and the sample labels, resulting in a preliminary model with some detection capability.
[0057] return Figure 1 In step S130, the violation detection model obtained from the first training is subjected to a second training based on the labeled samples and the unlabeled samples. In the second training, at least one perturbation information is added, which is used to reduce the gap between the output of the violation detection model under the unperturbed condition and the output of the violation detection model under the perturbed condition.
[0058] In some embodiments, the violation detection model obtained from the first training can be trained a second time through the following steps 1)-5).
[0059] In step 1), based on the baseline image and the violation image in the training image samples, the violation detection model is used to detect violation events in the violation image to obtain the first output image.
[0060] Taking a violation detection model that includes an encoder, a feature difference module, and a decoder as an example, step 1 above can be implemented in the following way.
[0061] First, based on the reference image and the violation image in the training image samples, the encoder is used to determine the first feature corresponding to the reference image and the second feature corresponding to the violation image, respectively.
[0062] Secondly, based on the first and second features of the training image samples, the difference features are determined using the feature difference module.
[0063] Finally, based on the differential features corresponding to the training image samples, the decoder is used to detect violations in the violation images, resulting in the first output image. This first output image includes both the first output image corresponding to the labeled samples and the first output image corresponding to the unlabeled samples.
[0064] For example, the processing procedure for unlabeled samples in step 1) above can be referred to Figure 3 The processing procedure for labeled samples will not be elaborated here.
[0065] In step 2), for the unlabeled sample, based on the corresponding reference image and violation image, while adding the at least one perturbation information, the violation detection model is used to detect violation events in the violation image to obtain a second output image.
[0066] Taking a violation detection model that includes an encoder, a feature difference module, and a decoder as an example, step 2 above can be implemented in the following way.
[0067] As described in the preceding embodiments, differential features corresponding to unlabeled samples can be obtained. Then, for the unlabeled samples, at least one perturbation information is added to the corresponding differential features to obtain perturbed differential features. Furthermore, based on the perturbed differential features corresponding to the unlabeled samples, a decoder is used to detect violations in the violation image, resulting in a second output image. For example, the differential features of the unlabeled samples are represented as follows:
[0068] Figure 4 This is a partial schematic diagram illustrating a second training process according to some embodiments of the present disclosure.
[0069] like Figure 4 As shown, the difference features of unlabeled samples The data is input to the decoder to obtain the raw output (the first output image). For example, the difference features of unlabeled samples. After undergoing at least one of perturbations 1 to 5, the perturbed differential features are obtained. At least one of the following. Difference characteristics after perturbation. At least one of them is input into the decoder to obtain at least one of perturbation output (second output image) 1 to 5.
[0070] In some embodiments, adding the at least one perturbation information to the corresponding differential features includes at least one of the following: adding random noise to the corresponding differential features, wherein the random noise has the same dimension as the differential features; randomly removing features in the corresponding differential features whose normalized feature values are less than a pre-configured feature threshold; using a random erasure algorithm to erase features in the corresponding differential features corresponding to the salient regions of the first output image corresponding to the unlabeled samples; performing mask operations on the corresponding differential features based on the salient regions of the first output image corresponding to the unlabeled samples; and adding virtual adversarial perturbations to the corresponding differential features.
[0071] In some embodiments, the difference features for the i-th pair of unlabeled samples (reference image and violation image) are: It is possible Add at least one of the following five types of perturbations.
[0072] Perturbation 1: Add random noise. Generate a dimension with... The same noise N. Each element of N ~ U(-0.3, 0.3). The perturbation characteristics are as follows:
[0073] Perturbation 2: Randomly remove some difference features with large responses. First, normalize the difference features: Then, a threshold t ~ U(0.6, 0.9) is randomly sampled or selected, and a mask is generated using a binary function based on this threshold t. The characteristics after the disturbance are The mask is a matrix with the same size as the difference features. The mask is generated by a binary function, so each element is either 0 or 1. A value of 1 is used when the normalized feature is less than a threshold t, and 0 otherwise. Therefore, after the dot product operation, the difference features at the corresponding positions are removed using the elements of the mask that are 0.
[0074] Perturbation 3: Feature erasure based on saliency map. To reduce the output's dependence on a certain feature, based on the saliency region in the output, we return to the corresponding position in the difference features and erase the feature at that position.
[0075] The saliency region / A of the output graph can be solved by channel: That is, by comparing the pixel values of each pixel in each channel of the output image (which contains two channels), the channel number containing the larger pixel value is selected. Therefore, the resulting saliency map SA is a binarized image.
[0076] Calculate the area of each connected region in the saliency map. For regions with an area greater than 1, use interpolation to find the region corresponding to the difference feature. Combined with the Random Erasing algorithm, randomly select a rectangular area within the difference feature region and set all feature points within the rectangular area to random values.
[0077] Perturbation 4: Feature discarding based on the saliency map. Masking is performed using both saliency and non-saliency regions, resulting in perturbed features. or That is, based on the saliency map SA, select the salient regions in the overall retained differential features, or select the non-salient regions in the overall retained differential features.
[0078] Perturbation 5: Virtual adversarial perturbation. Under the constraint that the vector magnitude does not exceed ∈, find a direction r that maximizes the change in output. vat . Where O is the model output graph, and f is the difference feature. These are the model parameters, and D measures the divergence between the two distributions. The perturbation characteristics are:
[0079] In step 3), for the labeled sample, the loss value of the first loss function is determined based on the corresponding first output image and label image. This loss value characterizes the degree of difference between the first output image and the label image corresponding to the labeled sample. For example, the loss value of the first loss function is characterized as L. l .
[0080] In step 4), for the unlabeled sample, the loss value of the second loss function is determined based on the corresponding first and second output images. This loss value characterizes the degree of difference between the first and second output images corresponding to the unlabeled sample. The loss value of the second loss function is applied to the first and second output images corresponding to the unlabeled sample as follows: Figure 4 The consistency constraints are shown. For example, the loss value of the second loss function is represented as L. ul .
[0081] By employing various random perturbation methods and coordinating the weighting and consistency constraints of two loss functions to address the differential features of the model, the model can produce robust and accurate detection results through training on the perturbation features.
[0082] In step 5), the violation detection model obtained from the first training is trained a second time based on the loss value of the first loss function and the loss value of the second loss function.
[0083] In the above embodiments, the second training is, for example, model deep training. During model deep training, unlabeled samples D are added. ul They jointly participated in model training. Due to the insufficient amount of labeled data, the output of the initial trained model was highly susceptible to the difference feature f. d The impact of perturbation. Therefore, in this deep training phase, perturbations are actively added to the differential features based on unlabeled samples, and the original output and the perturbed output are constrained to be as consistent as possible to optimize the model.
[0084] In some embodiments, the second training of the violation detection model obtained from the first training, based on the loss value of the first loss function and the loss value of the second loss function, includes the following steps.
[0085] First, the loss value of the second loss function is weighted according to a regularization parameter related to the number of iterations, where the regularization parameter is positively correlated with the number of iterations. For example, the regularization parameter is denoted as α(t), where t represents the number of iterations, and α(t) ranges from 0 to 1. When t is greater than T, α(t) is always equal to 1.
[0086] α(t) represents the basic function of the model in the early iterations, where labeled samples have higher weights and the model utilizes supervised information to quickly detect changes. As the number of iterations increases, the weights of unlabeled samples gradually increase, and the model uses unsupervised information to improve detection accuracy.
[0087] Then, the total loss value is determined by summing the loss value of the first loss function and the loss value of the weighted second loss function. When multiple perturbation information is added, there are multiple loss values for the second loss function. For example, the total loss value L... total =L l +α(t)I ul Among them, in the presence of multiple perturbations, I ul It is the sum of the loss values of multiple second loss functions.
[0088] Finally, based on the total loss value, the violation detection model obtained from the first training is trained a second time.
[0089] In some embodiments, the first loss function is the cross-entropy loss function, and the second loss function is the mean squared error loss function.
[0090] In the above embodiments, a preliminary model with certain detection capabilities is obtained through the first training. Through the second training, perturbation information is used to impose consistency constraints on the second training process, ensuring that the output of the violation event detection model under the unperturbed condition is as consistent as possible with the output of the violation event detection model under the perturbed condition, thereby optimizing the model. By combining a relatively small number of labeled samples with a relatively large number of unlabeled samples, consistency constraints are added to the training process through perturbation, enabling model training with a small number of labeled samples and reducing the cost of model development. Adding perturbation improves the model's robustness, generalization, and transferability.
[0091] In this disclosure, during deep training of the model, the proportion of two loss functions is coordinated according to the number of iterations, and various random perturbations are designed for the difference features of unlabeled sample pairs. By constraining the consistency between the original output and the output after random perturbation, the model is optimized. Since no assumptions are made about the distribution of unlabeled samples, the model can have better generalization and transferability when tested on another unseen dataset.
[0092] Figure 5 This is a flowchart illustrating a violation detection method according to some embodiments of the present disclosure.
[0093] like Figure 5 As shown, the violation detection method includes steps S510-S520.
[0094] In step S510, the image to be detected of the target scene is acquired.
[0095] In step S520, a violation detection model is used to detect violation events in the image to be detected, and an output image is obtained. The violation detection model is obtained through the training method in any of the foregoing embodiments.
[0096] In some embodiments, taking multiple images to be detected as an example, violations are accumulated based on the time difference between the multiple images to be detected and the output image; if the accumulated violations exceed a preset time threshold T, it is determined that there are violations in the target scene and an alarm is issued.
[0097] Multiple images to be detected are images from the same location area at different times. For example, for a specific area of interest, such as a fire lane marked by a road surface, monitoring images from multiple consecutive or discontinuous time periods can be acquired as multiple images to be detected. In the case of discontinuous images, the time interval between adjacent images should be less than a preset threshold to avoid misjudging the duration of violations.
[0098] Figure 6 This is a block diagram illustrating a training apparatus for a violation detection model according to some embodiments of the present disclosure.
[0099] like Figure 6 As shown, the training device 6 for the violation detection model includes an acquisition module 61, a first training module 62, and a second training module 63.
[0100] The acquisition module 61 is configured to acquire training image samples, for example, by performing the following: Figure 1 The step S110 is shown. The training image samples include a first sample size of labeled samples and a second sample size of unlabeled samples. Both the labeled and unlabeled samples include a baseline image without violations and a violation image with violations. The labeled samples also include label images describing the violations, with the first sample size being smaller than the second sample size. The baseline image and the violation image are used to describe the same location region.
[0101] The first training module 62 is configured to perform a first training on a violation detection model for detecting violations based on the labeled samples, for example, by performing the following: Figure 1 The step S120 shown.
[0102] The second training module 63 is configured to perform a second training on the violation detection model obtained from the first training, based on the labeled samples and the unlabeled samples, for example, by performing the following: Figure 1 Step S130 is shown. In the second training, at least one perturbation information is added, which is used to reduce the gap between the output of the violation detection model under the unperturbed condition and the output of the violation detection model under the perturbed condition during the second training.
[0103] Figure 7 This is a block diagram illustrating a violation detection apparatus according to some embodiments of the present disclosure.
[0104] like Figure 7 As shown, the violation detection device 7 includes an acquisition module 71 and a detection module 72.
[0105] The acquisition module 71 is configured to acquire the image to be detected, for example, by performing the following: Figure 5 The step S510 shown.
[0106] Detection module 72 is configured to use a violation detection model to detect violation events in the image to be detected and obtain an output image, for example, by performing... Figure 5 The step S520 is shown. The violation detection model is obtained through the training method in any embodiment of this disclosure.
[0107] Figure 8 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure.
[0108] like Figure 8 As shown, the electronic device 8 includes a memory 81 and a processor 82 coupled to the memory 81. The memory 81 is used to store instructions for executing embodiments of training methods or violation detection methods. The processor 82 is configured to execute training methods or violation detection methods in any of the embodiments of this disclosure based on the instructions stored in the memory 81.
[0109] Figure 9 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.
[0110] like Figure 9 As shown, the computer system 90 can be represented in the form of a general computing device. The computer system 90 includes a memory 910, a processor 920, and a bus 900 connecting different system components.
[0111] The memory 910 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for executing corresponding embodiments of at least one of the training method and the violation detection method. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.
[0112] The processor 920 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the decision module and the determination module, can be implemented by executing instructions in the central processing unit (CPU) memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.
[0113] Bus 900 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.
[0114] The computer system 90 may also include an input / output interface 930, a network interface 940, and a storage interface 950. These interfaces 930, 940, and 950, as well as the memory 910 and processor 920, can be connected via a bus 900. The input / output interface 930 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 940 provides a connection interface for various networked devices. The storage interface 950 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.
[0115] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.
[0116] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.
[0117] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.
[0118] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.
[0119] The training method and apparatus for violation detection models, the violation detection method and apparatus, the electronic equipment and the computer storage medium in the above embodiments can reduce the cost of model development and improve the generalization and transferability of the model.
[0120] The training method and apparatus for the violation detection model according to this disclosure, the violation detection method and apparatus, the electronic equipment, and the computer-storable medium have been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.
Claims
1. A training method for a violation detection model, comprising: Acquire training image samples, wherein the training image samples include a first sample size of labeled samples and a second sample size of unlabeled samples. Both the labeled samples and the unlabeled samples include a baseline image without violations and a violation image with violations. The labeled samples also include a label image describing the violation. The first sample size is smaller than the second sample size. The baseline image and the violation image are used to describe the same location area. Based on the labeled samples, a violation detection model for detecting violations is subjected to a first training, including: using an encoder to determine a first feature corresponding to the reference image and a second feature corresponding to the violation image based on the reference image and the violation image in the labeled samples; using a feature difference module to determine a difference feature based on the first feature and the second feature; using a decoder to detect violations in the violation image based on the difference feature corresponding to the labeled samples to obtain a first output image; determining the loss value of a first loss function based on the first output image and the label image of the labeled samples; and performing a first training on the violation detection model based on the loss value of the first loss function, wherein the violation detection model includes the encoder, the feature difference module, and the decoder; Based on the labeled samples and the unlabeled samples, the violation detection model obtained from the first training is subjected to a second training. In the second training, at least one perturbation information is added to the differential features of the unlabeled samples. The perturbation information is used to reduce the gap between the output of the violation detection model under the unperturbed condition and the output of the violation detection model under the perturbed condition in the second training.
2. The training method according to claim 1, wherein, The second training of the violation detection model obtained from the first training includes: Based on the baseline image and the violation image in the training image samples, the violation detection model is used to detect violation events in the violation image to obtain the first output image; For the unlabeled sample, based on the corresponding baseline image and the violation image, while adding at least one perturbation information, the violation detection model is used to detect violation events in the violation image to obtain a second output image; For the labeled sample, the loss value of the first loss function is determined based on the corresponding first output image and label image, wherein the loss value characterizes the degree of difference between the corresponding first output image and label image; For the unlabeled sample, the loss value of the second loss function is determined based on the corresponding first output image and second output image, wherein the loss value characterizes the degree of difference between the corresponding first output image and second output image; Based on the loss values of the first loss function and the second loss function, the violation detection model obtained from the first training is then trained a second time.
3. The training method according to claim 2, wherein, Based on the corresponding baseline image and the violation image, while adding the at least one perturbation information, the violation detection model is used to detect violation events in the violation image, resulting in a second output image including: Based on the reference image and the violation image in the training image samples, the encoder is used to determine the first feature corresponding to the reference image and the second feature corresponding to the violation image, respectively. Based on the first and second features of the training image samples, the difference features are determined using the feature difference module. For the unlabeled sample, at least one perturbation information is added to the corresponding difference feature to obtain the perturbed difference feature; Based on the perturbed differential features corresponding to the unlabeled samples, a decoder is used to detect illegal events in the illegal image and obtain a second output image.
4. The training method according to claim 3, wherein, Adding at least one perturbation to the corresponding difference feature includes at least one of the following: For the corresponding difference features, random noise is added, wherein the random noise has the same dimension as the difference features; Randomly remove features whose normalized feature values are less than the pre-configured feature threshold from the corresponding difference features; Based on the salient regions of the first output image corresponding to the unlabeled samples, a random erasure algorithm is used to erase the features corresponding to the salient regions in the differential features. Based on the salient regions of the first output image corresponding to the unlabeled samples, a mask operation is performed on the corresponding difference features; and Virtual adversarial perturbations are added to the corresponding differential features.
5. The training method according to claim 3, wherein, Based on the baseline image and the violation image in the training image samples, the violation detection model is used to detect violation events in the violation image, resulting in a first output image including: Based on the differential features corresponding to the training image samples, the decoder is used to detect illegal events in the illegal images and obtain the first output image.
6. The training method according to claim 2, wherein, Based on the loss values of the first loss function and the second loss function, the violation detection model obtained from the first training is then subjected to second training, including: The loss value of the second loss function is weighted according to a regularization parameter related to the number of iterations, wherein the regularization parameter is positively correlated with the number of iterations; The total loss value is determined by the sum of the loss value of the first loss function and the loss value of the weighted second loss function. In the case of adding multiple perturbation information, there are multiple loss values of the second loss function. Based on the total loss value, the violation detection model obtained from the first training is then trained a second time.
7. The training method according to claim 2, wherein, The first loss function is the cross-entropy loss function, and the second loss function is the mean squared error loss function.
8. The training method according to claim 1, wherein, The loss value of the first loss function characterizes the degree of difference between the first output image and the label image corresponding to the labeled sample.
9. The training method according to claim 1, wherein, The labeled samples and the unlabeled samples also include regions of interest that match the violation scenario. These regions of interest are part of the location region and are the core region where the violation event occurs in the violation scenario.
10. The training method according to claim 1, wherein, The ratio of the first sample size to the second sample size is less than or equal to 1 / 9.
11. A method for detecting violations, comprising: Acquire the image to be detected of the target scene; The violation detection model is used to detect violation events in the image to be detected, and an output image is obtained, wherein the violation detection model is obtained by the training method as described in claims 1 to 10.
12. The violation detection method according to claim 11, wherein, There are multiple images to be detected, and the violation detection method further includes: The violations are accumulated based on the time difference between multiple images to be detected and the output image; If the accumulated violations exceed a preset time threshold, a violation is determined to exist in the target scenario, and an alarm is issued.
13. A training device for a violation detection model, comprising: The acquisition module is configured to acquire training image samples, wherein the training image samples include a first sample size of labeled samples and a second sample size of unlabeled samples. Both the labeled samples and the unlabeled samples include a baseline image without violations and a violation image with violations. The labeled samples also include a label image describing the violation. The first sample size is smaller than the second sample size. The baseline image and the violation image are used to describe the same location area. The first training module is configured to perform a first training on a violation detection model for detecting violations based on the labeled samples. This includes: determining a first feature corresponding to the baseline image and a second feature corresponding to the violation image using an encoder, based on the baseline image and the violation image in the labeled samples; determining a difference feature using a feature difference module based on the first feature and the second feature; detecting violations in the violation image using a decoder based on the difference feature corresponding to the labeled samples to obtain a first output image; determining the loss value of a first loss function based on the first output image and the label image of the labeled samples; and performing a first training on the violation detection model based on the loss value of the first loss function. The violation detection model includes the encoder, the feature difference module, and the decoder. The second training module is configured to perform a second training on the violation detection model obtained from the first training based on the labeled samples and the unlabeled samples. In the second training, at least one perturbation information is added to the differential features of the unlabeled samples. The perturbation information is used to reduce the gap between the output of the violation event detection model under the unperturbed condition and the output of the violation event detection model under the perturbed condition in the second training.
14. A violation detection device, comprising: The acquisition module is configured to acquire the image to be detected; The detection module is configured to use a violation detection model to detect violation events in the image to be detected and obtain an output image, wherein the violation detection model is obtained by the training method as described in claims 1 to 10.
15. An electronic device comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the training method as described in any one of claims 1 to 10 or the violation detection method as described in any one of claims 11 to 12, based on instructions stored in the memory.
16. A computer-storeable medium having stored thereon computer program instructions that, when executed by a processor, implement the training method as claimed in any one of claims 1 to 10 or the violation detection method as claimed in any one of claims 11 to 12.