Regional early warning method, early warning equipment, electronic equipment and medium

Abstract

The present application discloses a regional early warning method, early warning equipment, electronic equipment and medium, the early warning method comprising: obtaining a background image to obtain a region to be compared of the background image; obtaining a foreground image of a target to be detected to obtain a target region of the foreground image, wherein the background image is a reference frame of the foreground image; based on the region to be compared, processing the target region using a difference detection model to determine multiple difference regions between the region to be compared and the target region; and issuing early warnings for multiple difference regions that meet the alarm rules. Through the above-mentioned method, the present application uses the difference detection model to focus on processing the target region, thereby finding multiple difference regions of the production line in the monitoring scene, and then issuing effective early warnings for the difference regions that meet the alarm rules.

Description

Technical Field

[0001] The present application relates to the field of image processing and computer vision technology, and in particular to a regional early warning method, early warning equipment, electronic equipment and medium. Background Art

[0002] With the development of deep learning and computer vision technology, it has become an urgent need to use video surveillance systems in factory production lines, conveyor belts and other areas to automatically detect whether machines are faulty and to provide timely warnings of abnormal fault conditions.

[0003] At present, some automatic detection research uses complex data for training to build machine learning models, and then uses the machine learning models to detect the collected image sequences. They often use empirical values ​​to judge anomalies or faults on the production line based on specific color space changes in the image. This leads to an increase in the proportion of subjective factors, thereby ignoring some anomalies, or easily making wrong judgments in the presence of interference. Summary of the Invention

[0004] In order to solve the above technical problems, the technical solution adopted in the first aspect of this application is to provide a regional early warning method, which includes: obtaining a background image to obtain the area to be compared of the background image; obtaining a foreground image of the target to be detected to obtain the target area of ​​the foreground image, and the background image is a reference frame of the foreground image; based on the area to be compared, the target area is processed using a difference detection model to determine multiple difference areas between the area to be compared and the target area; and issuing early warnings for multiple difference areas that meet the alarm rules.

[0005] In order to solve the above technical problems, the technical solution adopted in the second aspect of this application is to provide an early warning device, which includes:

[0006] An acquisition module is used to acquire a background image and obtain a region to be compared in the background image;

[0007] The acquisition module is further used to acquire a foreground image of the target to be detected, obtain a target area of ​​the foreground image, and the background image is a reference frame of the foreground image;

[0008] a difference detection module, configured to process the target region based on the region to be compared, and determine a plurality of difference regions between the region to be compared and the target region;

[0009] The early warning module is used to issue early warnings for multiple difference areas that meet the alarm rules.

[0010] In order to solve the above technical problems, the technical solution adopted in the third aspect of this application is to provide an electronic device, which includes: a processor and a memory, in which a computer program is stored, and the processor is used to execute the computer program to implement the early warning method described in the first aspect of this application.

[0011] In order to solve the above technical problems, the technical solution adopted in the fourth aspect of this application is to provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the early warning method described in the first aspect of this application.

[0012] The beneficial effects of the present application are as follows: the present application is based on the monitoring video of the production line, adopts image processing and computer vision technology, compares the monitoring video with the normal operating state based on the area to be compared, and focuses on the target area by using the difference detection model, thereby finding multiple difference areas of the production line in the monitoring scene, and then effectively warning the difference areas that meet the alarm rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] Figure 1 This is a flow chart of an embodiment of the regional early warning method of the present application;

[0015] Figure 2 This is a schematic diagram of the overall structure of a specific embodiment of the early warning method of the present application;

[0016] Figure 3 This application Figure 1 A flow chart of a specific embodiment of step S12;

[0017] Figure 4 This application Figure 1 A flow chart of a specific embodiment of step S13;

[0018] Figure 5 This is a schematic diagram of the structure of the attention mechanism module added to the bottleneck module of this application;

[0019] Figure 6 This application Figure 4 A flow chart of a specific embodiment of step S32;

[0020] Figure 7 This is a specific structural diagram of the difference detection module of this application;

[0021] Figure 8 This is a structural diagram of the connection between image feature extraction and difference feature enhancement in this application;

[0022] Figure 9This is a specific structural diagram of the difference feature enhancement module of this application;

[0023] Figure 10 This is a schematic diagram of a specific structure of the detection head of this application;

[0024] Figure 11 This is a flowchart of an embodiment of setting warning rules in this application;

[0025] Figure 12 This is a flowchart of an embodiment of the present application for performing an early warning according to early warning rules;

[0026] Figure 13 This is a flowchart of a specific embodiment of the present application for performing an early warning according to early warning rules;

[0027] Figure 14 This is a schematic block diagram of the structure of an embodiment of the early warning device of the present application;

[0028] Figure 15 This is a schematic block diagram of the structure of an electronic device embodiment of the present application;

[0029] Figure 16 This is a schematic block diagram of a circuit of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0031] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0032] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0033] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0035] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] In order to illustrate the technical solution of the present application, the following is an explanation through specific embodiments. The present application provides a regional early warning method, which can be applied to the early warning system of production line abnormalities, and can also be applied to the early warning system of detecting difference areas. The present application takes the early warning system of textile factory production line abnormalities based on monitoring video as an example, please refer to Figure 1 and Figure 2 , Figure 1 This is a flow chart of an embodiment of the regional early warning method of the present application. Figure 2 This is a schematic diagram of the overall structure of a specific embodiment of the early warning method of the present application, which specifically includes the following steps:

[0037] S11: Obtain a background image and obtain a region to be compared in the background image;

[0038] Usually, in order to detect the difference areas between two images, interference factors can often be eliminated. For example, to filter out the interference factors in the two images, a background can be pre-set as the background image of the difference detection model, and a fixed area can be defined as the main comparison area to provide a reliable comparison range for the subsequent comparison.

[0039] Among them, the background image is a static image, and the area to be compared can be circled manually. The specific selection can be made according to the actual situation and is not limited here.

[0040] S12: Obtain a foreground image of the target to be detected, and obtain a target area of ​​the foreground image;

[0041] Since the images being detected are often moving, when acquiring the foreground image of the target to be detected, a sequence of foreground images is often obtained. To obtain a fixed target area in the foreground image, a frame in the sequence of foreground images can be selected. The background image is the reference frame of the foreground image. Specifically, the background image and the foreground image are the background and target captured for the same scene. By using the background image as the reference frame, the differences between the foreground image and the background image can be found when comparing them.

[0042] Then, through specific analysis and processing of the foreground image, the target area of ​​the foreground image can be obtained. The detection results of the targets to be detected often include targets such as people, motor vehicles and non-motor vehicles, and image recognition can also determine the category of the targets to be detected and the coordinate information of each target to be detected in the foreground image.

[0043] Tracking the detected video footage can reveal the correlation between the movements of multiple targets to be detected, such as the path of a person walking, the avoidance zone between a vehicle and a person, and the category labels and information maps of each target to be detected during their movement. This allows the video motion information to be used as an aid to difference judgment, making it more conducive to filtering out invalid differences between the target area and the target area. S13: Based on the target area to be compared, the target area is processed using a difference detection model to determine multiple difference areas between the target area and the target area.

[0044] In order to determine the difference between the background image and the foreground image, a difference detection model can be constructed. Specifically, the Darknet53 network module in the field of target detection can be used to model the difference features in the background image and the difference features in the foreground image.

[0045] In this way, based on the area to be compared, the target area is processed by the difference detection model, so that multiple difference areas between the area to be compared and the target area can be obtained, thereby determining the multiple difference areas between the area to be compared and the target area.

[0046] Furthermore, with the auxiliary effect of video motion information, invalid differences can be eliminated based on multiple difference areas to obtain valid differences.

[0047] S14: issuing an early warning for multiple difference areas that meet the alarm rules.

[0048] Because it can be seen in step S13 that multiple valid differences are actually tracked and recorded in the end, alarm rules can be set. For example, if the difference area exceeds 10% of the preset threshold, it is considered a valid difference, or if a fixed target appears in the difference area, it is considered a valid difference. Therefore, multiple difference areas that meet the alarm rules are represented as valid differences. Therefore, when multiple valid difference areas do exist, an early warning is issued. Therefore, unnecessary early warning reminders are avoided and early warning resources can be effectively saved.

[0049] Therefore, this application is based on the monitoring video of the production line, and adopts image processing and computer vision technology. Based on the area to be compared, the monitoring video is compared with the normal operating state, and the target area is focused on by using the difference detection model to find out the multiple difference areas that appear in the production line in the monitoring scene, and then effectively warn the difference areas that meet the alarm rules.

[0050] Among them, the steps of obtaining the foreground image of the target to be detected and obtaining the target area of ​​the foreground image are as follows: Figure 3 , Figure 3 This application Figure 1 The flowchart of step S12 in a specific embodiment specifically includes the following steps:

[0051] S21: Acquire a foreground image at intervals of a preset number of frames;

[0052] Specifically, in order to effectively eliminate a series of interference factors such as lighting, jitter, and non-target movement of humans and machines in the subsequent process, and because the actual situation is to monitor the production line in real time, a huge storage space is required when obtaining the foreground image sequence.

[0053] However, the actual storage space is limited, so it is possible to obtain foreground images of a preset number of frames at intervals of time. On the one hand, this can save storage space resources, and on the other hand, it can accurately eliminate some interference factors, making it easier to detect the operating status of the production line.

[0054] S22: Detecting the target to be detected in the foreground image based on the target detection model to obtain target information of the foreground image, where the target information at least includes a detection frame and a target category of the target to be detected;

[0055] Generally speaking, for the processing of foreground image sequences, please refer to Figure 2 In the overall structural diagram, a target detection model is often set up in the overall system to process the acquired foreground image in real time.

[0056] By obtaining a foreground image with a preset number of frames at intervals, the target detection model is input. Since there may be non-interesting targets (such as people) in the foreground image, the target detection model can detect such targets in the foreground image for filtering out in the later warning. The target information includes at least the detection box and target category of the target to be detected.

[0057] S23: According to the target category, the detection frame of the target to be detected is calibrated to obtain the target area.

[0058] Specifically, after knowing the category of the target to be detected, we can know whether the target to be detected is a person or a car, etc. By associating the target category with the target to be detected, we can perform regional calibration on the detection frame of the target to be detected for filtering in the later warning.

[0059] Therefore, when performing motion tracking processing on a target to be detected, the motion trajectory area of ​​the target area is often obtained first, and then the motion trajectory of the target to be detected can be determined by finding the center point of the target area.

[0060] Furthermore, based on the region to be compared, the target region is processed using a difference detection model to determine a plurality of difference regions between the region to be compared and the target region. Figure 4 , Figure 4 This application Figure 1 The flowchart of step S13 in a specific embodiment specifically includes the following steps:

[0061] S31: Using the attention mechanism module in the target detection model to extract image features from the foreground image and the background image respectively, to obtain first multi-level difference features of the area to be compared and second multi-level difference features of the target area;

[0062] The object detection model at least includes an attention mechanism module, specifically the Convolutional Block Attention Model (CBAM). Given an input feature map, the CBAM module sequentially infers attention maps along two independent dimensions (i.e., channel and spatial), and then multiplies the attention map with the input feature map for adaptive feature optimization.

[0063] Therefore, by using the attention mechanism module to extract image features of the foreground image and the background image respectively, the first multi-level difference features of the area to be compared and the second multi-level difference features of the target area are obtained, wherein the multi-level difference features include at least image channel difference features and image space difference features.

[0064] S32: using a difference feature enhancement module of the difference detection model, performing feature enhancement on the first multi-level difference features and the second multi-level difference features, respectively, to obtain a first enhanced feature map corresponding to the foreground image and a second enhanced feature map corresponding to the background image;

[0065] Among them, the difference detection model of the early warning system also includes a difference feature enhancement module, and the difference feature enhancement module can highlight the areas with differences and weaken the areas without differences. In this way, the first multi-level difference features and the second multi-level difference features are enhanced respectively, thereby obtaining the first enhanced feature map corresponding to the foreground image and the second enhanced feature map corresponding to the background image.

[0066] S33: Inputting the first enhanced feature map and the second enhanced feature map into the channel attention module for further enhancement to establish a connection between the difference features between the foreground image and the background image;

[0067] Among them, the early warning system also includes a channel attention module (Squeeze-and-Excitation, SE), which focuses on channel information and can solve the loss problem caused by the different importance of feature maps in different channels during the convolution pooling process.

[0068] Therefore, the first enhanced feature map and the second enhanced feature map are input into the channel attention module for further enhancement to establish a connection between the difference features between the foreground image and the background image, so that they can adaptively allocate channels to each feature image during the enhanced fusion process, thereby improving the learning ability of the difference feature enhancement model.

[0069] S34: Merging the difference features with the multi-level network features to obtain multiple difference regions.

[0070] Specifically, the early warning system is equipped with a Feature Pyramid Network (FPN) structure. The FPN structure merges the difference features with the multi-level network features, calculates the similarity between the obtained multiple difference image features and the input image features, and obtains a similarity map with a channel number of 1, so that subsequent regression and classification can be performed to obtain multiple difference areas.

[0071] Among them, the target detection module contains multiple backbone networks, which include bottleneck modules; and multiple backbone networks can be cascaded.

[0072] In addition, before using the attention mechanism module in the target detection model to extract image features from the foreground image and the background image to obtain the first multi-level difference features of the to-be-compared region and the second multi-level difference features of the target region, the early warning method further includes: embedding the attention mechanism module into the bottleneck module to obtain an updated bottleneck module. Figure 5 , Figure 5 This is a structural diagram of the attention mechanism module added to the bottleneck module of this application.

[0073] like Figure 5 As shown, the BN layer facilitates gradient updates. Integrating the BN layer into Conv is equivalent to modifying the convolution kernel without increasing the amount of convolution computation, speeding up network inference. To train deep neural networks, a ReLU (Rectified Linear Activation Function) activation function is required. It looks and behaves like a linear function, but is actually a nonlinear function that allows for learning complex relationships in the data. This function must also provide more sensitive activation and input to avoid saturation.

[0074] Among them, the attention mechanism module can extract the weights of the output features of the bottleneck module before the update to obtain the spatial attention weights of the image. The spatial attention weights are multiplied at the pixel level with the output features of the bottleneck module before the update, and are cross-layer linked with the input features of the image, so that the updated bottleneck module can obtain the multi-level first feature map corresponding to the output of the foreground image and the multi-level second feature map corresponding to the output of the background image.

[0075] Furthermore, the multi-level first feature map includes at least the first multi-level difference feature, and the multi-level second feature map includes at least the second multi-level difference feature; using the difference feature enhancement module of the difference detection model, the first multi-level difference feature and the second multi-level difference feature are respectively enhanced to obtain the first enhanced feature map corresponding to the foreground image and the second enhanced feature map corresponding to the background image, please refer to Figures 6 to 9 , Figure 6 This application Figure 4 The flowchart of step S32 in a specific embodiment is as follows: Figure 7 This is a specific structural diagram of the difference detection module of this application; Figure 8 This is a structural diagram of the connection between image feature extraction and difference feature enhancement in this application;

[0076] Figure 9 This is a schematic diagram of a specific structure of the difference feature enhancement module of this application, which specifically includes the following steps:

[0077] S41: performing convolution processing on the multi-level first feature map and the multi-level second feature map respectively using the convolution branch module stacked by the difference feature enhancement module;

[0078] like Figure 7 As shown in the figure, the difference feature enhancement module often includes stacked convolution branches, specifically a convolution branch composed of 3 layers of conv+ReLU. Through the stacked convolution branch module, the multi-level first feature map and the multi-level second feature map can be convolved respectively.

[0079] Image 1 is the foreground image, and Image 2 is the background image. Among C1-C5, only C3-C5 are used because the previous layers C1-C2 are relatively low, and C3-C5 focus more on semantic features. P3-P5 are from base-level features to joint features. Upsampling is a feedback mechanism. The detection head is a 5-layer convolutional layer structure, which is not described in detail here.

[0080] like Figure 8 As shown, the difference feature enhancement is actually the difference feature fusion, which is essentially the same processing operation. The residual block corresponds to Figure 5 The bottleneck module can highlight the areas with differences and weaken the areas without differences.

[0081] S42: performing pixel-level addition and subtraction on the output of the corresponding foreground image obtained after the convolution process, and taking the absolute value of the subtracted features to obtain a first enhanced feature map;

[0082] Specifically, if Figure 9 As shown in the figure, the output of the corresponding foreground image after convolution processing is divided into two paths. For the foreground image, one path is the output obtained by convolving the multi-level first feature map of the foreground image through the stacked convolution branch module; the other path is the output obtained by convolving the multi-level second feature map of the background image that shares the weight of the foreground image through the stacked convolution branch module. These two paths are added at the pixel level at each level, and the added outputs are subtracted, and the absolute value of the subtracted features is taken. Then, after a conv+BN+ReLU structure, the first enhanced feature map is obtained.

[0083] S43: Perform pixel-level addition on the output of the corresponding background image obtained after the convolution process to obtain a second enhanced feature map.

[0084] Specifically, if Figure 9As shown in the figure, the output of the corresponding background image after convolution processing is divided into two paths. For the background image, one path is the output obtained by convolving the multi-level second feature map of the background image through the stacked convolution branch module; the other path is the output obtained by convolving the multi-level first feature map of the foreground image that shares the weights with the background image through the stacked convolution branch module. These two paths are added at the pixel level at each level, and the added output is added again at the pixel level, and then passed through a conv+BN+ReLU structure to obtain the second enhanced feature map.

[0085] Among them, Conv+BN+Relu is a common model structure in mainstream convolutional neural network models. During model inference and training, the BN layer is often merged with other layers to reduce the amount of computation.

[0086] In addition, before you proceed with the steps for alerting multiple difference areas that meet the alert rules, refer to Figure 10 and Figure 11 , Figure 10 This is a schematic diagram of a specific structure of the detection head of this application. Figure 11 This is a flowchart of an embodiment of setting warning rules in the present application. The warning method also includes:

[0087] S51: Calculate the center point of the difference area and the bounding area around the center point to serve as a difference detection frame, which is used to traverse and detect the difference area;

[0088] In order to make full use of the feature information of the image pair, this application adopts the following method in the detection head part: Figure 10 The structure shown calculates the similarity of the backbone features of the input image pair to obtain a similarity map with a channel number of 1, performs a concat operation with the feature P output by the FPN structure, and then performs subsequent regression and classification.

[0089] That Figure 10 The prior frame in can be used as a difference detection frame. Specifically, the center point of the difference area and the bounding area around the center point are calculated to serve as the difference detection frame. The difference detection frame is used to traverse and detect the difference area.

[0090] S52: Setting a plurality of alarm rules according to the difference areas detected by traversal.

[0091] like Figure 11 As shown, the various alarm rules include:

[0092] Based on the object detection model, exclude the detected difference areas in the foreground image that meet the first preset threshold of the intersection over union ratio. Specifically, filter the target boxes in the target detection model output result rect_det that belong to human, motor vehicle, and non-motor vehicle types, and calculate the intersection over union ratio IoU_det between them and the difference area rect_i. If this value is greater than the set threshold thresh1, it is determined to be human, motor vehicle, and non-motor vehicle, and the difference area rect_i is not processed further.

[0093] Based on the target tracking model, exclude the difference areas in the tracked multi-frame foreground image that meet the second preset threshold of the intersection over union ratio. Specifically, filter the target boxes in the target tracking model output result rect_track that belong to human, motor vehicle, and non-motor vehicle types, and calculate the intersection over union ratio IoU_track between them and the difference area rect_i. If the value is greater than the set threshold thresh2, it is determined to be human or non-motor vehicle, and the difference area rect_i is no longer processed.

[0094] Based on multiple foreground image sequences, invalid regions where the ratio of the foreground image to the difference detection frame is greater than a third intersection preset threshold are filtered out. Specifically, motion region detection is performed on the video. For the difference region detection frame after human-machine non-filtering, the intersection between it and the motion region is calculated and its pixel value is counted. If the ratio of this pixel value to the difference region detection frame is greater than the preset threshold thresh3, the difference region is deemed invalid.

[0095] For more information on the steps to generate alerts for multiple difference areas that meet the alert rules, see Figure 12 , Figure 12 This is a flowchart of an embodiment of the present application for performing an early warning according to early warning rules, which specifically includes the following steps:

[0096] S61: Tracking and recording the difference areas that meet multiple alarm rules;

[0097] Specifically, the multiple alarm rules are consistent or similar to the multiple alarm rules set according to the difference areas detected by traversal in step S52, which will not be repeated here. If they are satisfied, the difference areas that meet the multiple alarm rules are tracked and recorded. If not, they are filtered out.

[0098] S62: If it is determined that the duration of the difference area exceeds a preset time period, an early warning is issued for the difference area.

[0099] See also Figure 13 , Figure 13 This is a flowchart of a specific embodiment of the present application for performing an early warning according to the early warning rules, which specifically includes the following steps:

[0100] S71: traverse the current frame difference detection box Rect_i and initialize t_i;

[0101] First, calculate the center point p of the difference region rect and calculate whether the center point is within the defined region region, and traverse the difference detection frame rect_i. When the movement range of the frame is smaller (compared to the previous detection frame), it is considered to be the same difference frame. At this time, t i =t i-1 Otherwise, it is considered to be a new difference frame, and t i =0.

[0102] S72: Obtain video motion information aera according to the target detection frame rect_det and the target tracking frame rect_track;

[0103] S73: Calculate IoU_det: flag1 = IoU_det > thresh1;

[0104] Specifically, the target boxes belonging to human, motor vehicle, and non-motor vehicle types in the output result rect_det of the target detection model are screened, and the intersection over union (IoU) ratio (IoU_det) between them and the difference area rect_i is calculated. When the value flag1 is greater than the set threshold thresh1, it is determined that it belongs to human, motor vehicle, and non-motor vehicle types, and the difference area rect_i will no longer be processed subsequently.

[0105] S74: Calculate IoU_track: flag2 = IoU_det > thresh2;

[0106] Specifically, the target frames belonging to human, motor vehicle, and non-motor vehicle types in the output result rect_track of the target tracking model are screened, and the intersection over union (IoU) ratio (IoU_track) between them and the difference area rect_i is calculated. When the value flag2 is greater than the set threshold thresh2, it is determined that it belongs to human, motor vehicle, and non-motor vehicle types, and the difference area rect_i is no longer processed subsequently.

[0107] S75: Calculate the pixel value ratio of the intersection area: flag3 = ratio > thresh3;

[0108] Specifically, motion area detection is performed on the video. For the difference area detection frame after human-machine non-filtering, its intersection with the motion area is calculated, and its pixel value is counted. When the ratio of the pixel value flag3 to the difference area detection frame is greater than the set threshold thresh3, the difference area is determined to be invalid.

[0109] S76: Determine flag1 & flag2 & flag3; if it is determined that the difference area is invalid, proceed to step S77, that is, calculate the duration t_i = t_i + 1.

[0110] S78: Determine whether the duration t of the difference area is greater than the detection period T;

[0111] The preset time period is the detection period T. If the duration of the difference region exceeds the preset time period, an early warning is issued for the difference region, and the process proceeds to step S79, where an alarm is output. If the duration of the difference region does not exceed the preset time period, the process proceeds to step S71, where the difference detection frame Rect_i of the current frame is traversed and t_i is initialized.

[0112] Therefore, directly based on image processing and computer vision technology, the problem of production line abnormality warning in key area monitoring scenarios can be solved quickly and efficiently.

[0113] In order to eliminate interference from lighting, shadows, jitter, etc., this application uses a deep learning difference detection model to enhance the feature information extraction of foreground and background image pairs by embedding a CBAM module in the bottleneck structure of the backbone network; this proposal considers the fusion problem of foreground and background difference features, and uses stacked convolutional layers and cross-layer cascades to perform pixel-level addition, subtraction and absolute value operations on feature maps as a difference feature enhancement structure to further improve the expression ability of difference features; at the same time, this application uses foreground and background features to measure similarity in the detection head to optimize the difference detection model.

[0114] Moreover, this application does not need to rely on other human experience and machine parameters, etc. It is directly based on image processing technology, and by designing alarm rules and comprehensively utilizing target detection and tracking results, it filters out non-major difference areas and filters according to motion information, which can more effectively provide early warning for changed areas.

[0115] In order to illustrate the technical solution of this application, this application also provides an early warning device, please refer to Figure 14 , Figure 14 : is a schematic block diagram of the structure of an embodiment of the early warning device of the present application, the early warning device 90 includes:

[0116] An acquisition module 91 is used to acquire a background image and obtain a region to be compared in the background image;

[0117] The acquisition module 91 is further used to acquire a foreground image of the target to be detected, and obtain a target area of ​​the foreground image, with the background image being a reference frame of the foreground image;

[0118] a difference detection module 92 for processing the target region based on the region to be compared, and determining a plurality of difference regions between the region to be compared and the target region;

[0119] The early warning module 93 is used to issue early warnings for multiple difference areas that meet the alarm rules.

[0120] Therefore, this application is based on the monitoring video of the production line, and adopts image processing and computer vision technology. Based on the area to be compared, the monitoring video is compared with the normal operating state, and the target area is focused on by using the difference detection model to find out the multiple difference areas that appear in the production line in the monitoring scene, and then effectively warn the difference areas that meet the alarm rules.

[0121] In order to illustrate the technical solution of the present application, the present application also provides an electronic device, which can be a computer or a mobile phone, etc., without specific limitation. Figure 15 , Figure 15 This is a schematic block diagram of the structure of an electronic device embodiment of the present application. The electronic device 100 includes: a processor 110 and a memory 120. The memory 120 stores a computer program 121. The processor 110 is used to execute the computer program 121 to implement the early warning method of the embodiment of the present application, which will not be repeated here.

[0122] In addition, this application also provides a computer-readable storage medium, see Figure 16 , Figure 16 This is a circuit schematic block diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 200 stores a computer program 201. When the computer program 201 is executed by a processor, it can implement the early warning method of the embodiment of the present application, which will not be repeated here.

[0123] If it is implemented in the form of a software functional unit and sold or used as an independent product, it can also be stored in a device with a storage function. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage device, including a number of instructions (program data) to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage device includes various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and electronic devices such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0124] The description of the execution process of program data in the device with storage function can be referred to the description in the above-mentioned method embodiment of the present application, which will not be repeated here.

[0125] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A regional early warning method, characterized in that: The early warning method includes: Obtaining a background image and obtaining a region to be compared in the background image; Acquire a foreground image of the target to be detected, and obtain a target area of ​​the foreground image, wherein the background image is a reference frame of the foreground image; Based on the area to be compared, the target area is processed using a difference detection model to determine a plurality of difference areas between the area to be compared and the target area; issuing an early warning for the plurality of difference areas that meet the alarm rules; The step of obtaining a foreground image of a target to be detected and obtaining a target area of ​​the foreground image includes: Acquire the foreground image at intervals of a preset number of frames; Based on the target detection model, detect the target to be detected in the foreground image to obtain target information of the foreground image, where the target information at least includes a detection frame and a target category of the target to be detected; According to the target category, the detection frame of the target to be detected is calibrated to obtain the target area.

2. The early warning method according to claim 1, characterized in that: The method of processing the target region by using a difference detection model based on the region to be compared to determine a plurality of difference regions between the region to be compared and the target region includes: Using the attention mechanism module in the target detection model to perform image feature extraction on the foreground image and the background image respectively, to obtain a first multi-level difference feature of the area to be compared and a second multi-level difference feature of the target area, wherein the multi-level difference feature includes at least an image channel difference feature and an image space difference feature; Using the difference feature enhancement module of the difference detection model, feature enhancement is performed on the first multi-level difference feature and the second multi-level difference feature to obtain a first enhanced feature map corresponding to the foreground image and a second enhanced feature map corresponding to the background image; Inputting the first enhanced feature map and the second enhanced feature map into a channel attention module for further enhancement, so as to establish a connection between the difference features between the foreground image and the background image; The difference features are combined with multi-level network features to obtain multiple difference regions.

3. The early warning method according to claim 2, characterized in that: The target detection model includes multiple backbone networks, and the backbone network includes a bottleneck module; Before extracting image features from the foreground image and the background image using the attention mechanism module in the target detection model to obtain first multi-level difference features of the to-be-compared area and second multi-level difference features of the target area, the early warning method further includes: Embedding the attention mechanism module into the bottleneck module to obtain an updated bottleneck module; Among them, the attention mechanism module can extract the weights of the output features of the bottleneck module before the update to obtain the spatial attention weights of the image. The spatial attention weights are multiplied at the pixel level with the output features of the bottleneck module before the update, and are cross-layer linked with the input features of the image, so that the updated bottleneck module can obtain the multi-level first feature map corresponding to the output of the foreground image and the multi-level second feature map corresponding to the output of the background image.

4. The early warning method according to claim 3, characterized in that: The multi-level first feature map includes at least the first multi-level difference feature, and the multi-level second feature map includes at least the second multi-level difference feature; The difference feature enhancement module using the difference detection model performs feature enhancement on the first multi-level difference feature and the second multi-level difference feature respectively to obtain a first enhanced feature map corresponding to the foreground image and a second enhanced feature map corresponding to the background image, including: Using the stacked convolution branch modules in the difference feature enhancement module to perform convolution processing on the multi-level first feature map and the multi-level second feature map respectively; Performing pixel-level addition and subtraction on the output corresponding to the foreground image obtained after the convolution processing, and taking the absolute value of the subtracted features to obtain the first enhanced feature map; The output corresponding to the background image obtained after the convolution process is added at the pixel level to obtain the second enhanced feature map.

5. The early warning method according to claim 4, characterized in that: Before issuing an early warning for the plurality of difference regions that meet the alarm rule, the early warning method further includes: Calculating a center point of the difference area and a bounded area around the center point as a difference detection frame, wherein the difference detection frame is used to traverse and detect the difference area; Multiple alarm rules are set according to the difference areas detected by traversal, wherein the multiple alarm rules include: Based on the target detection model, excluding the detected difference area in the foreground image that meets a first intersection-over-union preset threshold; Based on the target tracking model, excluding the difference area in the tracked multiple frames of the foreground image that meets the second intersection-over-union preset threshold; According to the plurality of foreground image sequences, invalid areas where the ratio of the foreground image to the difference detection frame is greater than a third cross preset threshold are filtered out.

6. The early warning method according to claim 5, characterized in that: The issuing of an early warning for the plurality of difference areas that meet the alarm rules includes: Track and record the difference areas that meet multiple alarm rules; If it is determined that the duration of the difference area exceeds a preset time period, an early warning is issued for the difference area.

7. An early warning device, characterized in that: The early warning equipment includes: An acquisition module is used to acquire a background image of the target to be detected and obtain a region to be compared in the background image; The acquisition module is further configured to acquire a foreground image of the target to be detected, and obtain a target area of ​​the foreground image, wherein the background image is a reference frame of the foreground image; The step of obtaining a foreground image of a target to be detected and obtaining a target area of ​​the foreground image includes: Acquire the foreground image at intervals of a preset number of frames; Based on the target detection model, detect the target to be detected in the foreground image to obtain target information of the foreground image, where the target information at least includes a detection frame and a target category of the target to be detected; According to the target category, the detection frame of the target to be detected is calibrated to obtain the target area; a difference detection module, configured to process the target area based on the area to be compared, and determine a plurality of difference areas between the area to be compared and the target area; The early warning module is used to issue an early warning for the plurality of difference areas that meet the alarm rules.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the early warning method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the early warning method according to any one of claims 1 to 6.

Images