Image processing method, device and terminal equipment based on distributed monitoring system
By using the object detection model and attention graph mechanism in a distributed monitoring system, combining local and global information to update confidence, the problem of high false alarm rate caused by global scanning in the existing technology is solved, and the specific targets in local scenarios are achieved. Accurate identification and efficient processing.
Patent Information
- Application Number
- CN202411266719.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-11
AI Technical Summary
The existing distributed monitoring system cannot accurately locate and identify specific targets in local scenes through global scanning and global feature extraction during image processing, resulting in a high false positive rate.
Using the image processing method based on the distributed monitoring system, the image to be processed in the monitoring area is obtained in real time, the object detection box is extracted using the preset object detection model and local and global attention maps are generated, and the confidence is updated in combination with the target degree value, so as to accurately Identify specific targets in key areas.
It reduces the waste of computing resources, improves image processing efficiency, reduces false alarm rates, can focus more accurately on key areas in the image, and accurately identify target objects, avoiding light and shadow changes and noise interference.
Smart Images

Figure CN118799814B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing of a monitoring system, and in particular to an image processing method, apparatus and terminal equipment based on a distributed monitoring system. Background Art
[0002] Distributed monitoring systems can use distributed computing resources and technologies to efficiently and quickly process and analyze image data in the monitoring system. Based on the characteristics of distributed systems, they can also decompose large-scale computing tasks (such as image processing) into multiple subtasks and execute them in parallel on multiple computing nodes, thereby significantly improving processing speed and efficiency.
[0003] When processing images in a distributed monitoring system, it usually refers to the collection, storage, processing and analysis of image data from multiple monitoring front ends (such as cameras) in a distributed computing environment. However, when collecting image data from multiple monitoring front ends (such as cameras) and identifying abnormal situations, the existing technology often adopts a global scanning method for images in each monitoring area, and then extracts significant features from each global image through image processing algorithms (such as edge detection, color analysis, texture analysis, etc.) or conventional image processing models to identify specific targets in the monitoring scene, such as people, vehicles, animals, etc., so that abnormalities can be detected in the monitoring area by analyzing specific targets in the monitoring scene.
[0004] However, global scanning and global feature extraction of the monitoring areas of all sub-node cameras will not only increase the computing resources in non-critical areas, resulting in low image processing efficiency, but also fail to accurately locate and identify specific targets in local scenes in critical areas. As a result, some non-critical changes or noises, such as light and shadow changes, swaying leaves, etc., may be misjudged as abnormal situations, resulting in a high false alarm rate. Summary of the invention
[0005] The embodiments of the present invention provide an image processing method, apparatus and terminal device based on a distributed monitoring system, construct a multi-level attention mechanism from local to global, which can accurately locate and identify specific targets in key areas of local scenes, and can effectively solve the problem in the prior art that only global scanning and global feature extraction are performed, which makes it impossible to accurately locate and identify specific targets in local scenes, resulting in a high false alarm rate.
[0006] An embodiment of the present invention provides an image processing method based on a distributed monitoring system, comprising:
[0007] Acquire the image to be processed corresponding to each monitoring area in real time;
[0008] For each image to be processed, the image to be processed is input into a preset target detection model, so that the target detection model extracts a number of target detection frames from the image to be processed, and generates an initial confidence level corresponding to each target detection frame; wherein the confidence level is used to characterize the reliability of whether a target object exists in the target detection frame;
[0009] Based on the image to be processed and each initial confidence level, a local attention map corresponding to each monitoring area is generated;
[0010] According to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map, the average attention value of the area corresponding to the target detection box in the global attention map is generated;
[0011] For each target detection frame, generate an updated confidence score corresponding to the target detection frame according to the target degree value, the regional average attention value, and the local attention map corresponding to the target detection frame; wherein the target degree value is used to characterize the degree of spatial and temporal association between the target detection frame and other target detection frames;
[0012] When it is determined that an updated confidence is greater than a preset confidence threshold, the corresponding target detection frame is used as a target frame to be warned, and an abnormal alarm result corresponding to the target frame to be warned is generated;
[0013] The training of the target detection model includes:
[0014] With several image samples as input and the actual target detection frame of each image sample and the corresponding actual confidence as output, the target detection model to be trained is iteratively trained until the model converges to generate a preset target detection model.
[0015] Preferably, the generating of a local attention map corresponding to each monitoring area based on the image to be processed and each initial confidence level includes:
[0016] Obtain a preset background image corresponding to the monitoring area at the previous moment, then compare the preset background image with the image to be processed, and generate a motion detection map according to the comparison result; wherein the motion detection map includes a background and a target in motion; the preset background image is a static image corresponding to the monitoring area;
[0017] Generate a current attention map corresponding to the current moment according to the historical local attention map corresponding to the previous moment, the historical attention map corresponding to the previous moment, and the attenuation factor; wherein the attenuation factor is used to characterize the contribution of historical information at the current moment;
[0018] According to each target detection frame, the initial confidence of each target detection frame, the motion detection map and the current attention map, a local attention map corresponding to each monitoring area is generated.
[0019] Preferably, a local attention map corresponding to each monitoring area is generated according to each target detection frame, the initial confidence of each target detection frame, the motion detection map and the current attention map, including:
[0020] The local attention map corresponding to each monitoring area is calculated according to the following formula:
[0021] ;
[0022] ;
[0023] ;
[0024] ;
[0025] in, is the local attention map corresponding to each monitoring area, The target detection box detection result corresponding to each monitoring area, is the initial confidence of the kth target detection box, is the binary mask of the kth target detection box, represents the sigmoid activation function, n is the number of target detection boxes; is the motion detection graph, is the image to be processed at the current moment, is the preset background image corresponding to the previous moment; is the current attention map corresponding to the current moment, is the historical local attention map corresponding to the previous moment of the monitoring area, is the historical attention map corresponding to the previous moment, is the attenuation factor; , are different weight coefficients, and + + =1.
[0026] Preferably, generating the average attention value of the region corresponding to the target detection frame in the global attention map according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map includes:
[0027] Generate a first global attention map at the current moment according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map;
[0028] Generate a second global attention map after time smoothing operation according to the historical global attention map at the previous moment, the first global attention map at the current moment, and a preset time smoothing coefficient;
[0029] Performing convolution calculation on the second global attention map to generate a third global attention map after spatial smoothing operation;
[0030] According to the coordinate information corresponding to the target detection box in the global coordinate system, an average attention value of the area corresponding to the target detection box in the third global attention map is generated.
[0031] Preferably, the target degree value includes: a first degree value and a second degree value;
[0032] The generating, according to the target degree value, the regional average attention value and the local attention map corresponding to the target detection frame, an updated confidence corresponding to the target detection frame comprises:
[0033] A monitoring area different from the monitoring area of the target detection frame is used as the target monitoring area, and then a first degree value is generated according to a determination result of whether there is a target detection frame identical to the target detection frame in the target monitoring area and a region weight value corresponding to each target monitoring area and used to characterize the size of the coverage area between two adjacent monitoring areas; wherein the first degree value is used to characterize the consistency degree of whether the target detection frame is detected consistently between different monitoring areas;
[0034] Extracting a number of historical moments according to a preset time interval; wherein each historical moment corresponds to a time weight value, and the smaller the time difference between a historical moment and the current moment, the larger the corresponding time weight value;
[0035] Generate a second degree value according to the target detection frame detection results corresponding to each historical moment and each time weight value; wherein the target detection frame detection result includes: detecting the target detection frame or not detecting the target detection frame; the second degree value is used to characterize whether the detection results corresponding to the target detection frame at different moments maintain the same degree of stability;
[0036] According to the first degree value, the second degree value, the regional average attention value and the local attention map corresponding to the target detection frame, an updated confidence corresponding to the target detection frame is generated.
[0037] Preferably, the generation of the first degree value includes:
[0038] The first degree value can be calculated according to the following formula:
[0039] ;
[0040] in, is the first degree value, is the regional weight value, is the target detection frame detection result corresponding to a target monitoring area, is the number of target monitoring areas.
[0041] Preferably, generating the second degree value according to the target detection box detection results corresponding to each historical moment and each time weight value includes:
[0042] The second degree value is calculated according to the following formula:
[0043] ;
[0044] in, is the second degree value, is the attenuation factor; is the time difference between the historical moment and the current moment, is the time weight value corresponding to each historical moment, Indicates the target detection frame detection result corresponding to the historical moment, =1 means the target detection frame is detected. =0 means that the target detection frame is not detected. is the number of historical moments.
[0045] Preferably, when iterative training is performed on the target detection model to be trained, the method further includes:
[0046] During each training, an image sample is input into the target detection model so that the target detection model generates a target detection box prediction result and a confidence prediction result corresponding to the image sample according to the image features of the image sample;
[0047] The target detection box prediction results and confidence prediction results are compared with the actual target detection box results and actual confidence results corresponding to the image samples, and then the network parameters of the target detection model are adjusted according to the comparison results.
[0048] Based on the above method embodiments, the present invention provides corresponding device embodiments.
[0049] An embodiment of the present invention provides an image processing device based on a distributed monitoring system, comprising: an image acquisition module, an image processing module, a local attention processing module, a global attention processing module, a confidence update module, and an abnormal alarm module;
[0050] The image acquisition module is used to acquire the image to be processed corresponding to each monitoring area in real time;
[0051] The image processing module is used to input each image to be processed into a preset target detection model, so that the target detection model extracts a plurality of target detection frames from the image to be processed and generates an initial confidence level corresponding to each target detection frame; wherein the confidence level is used to characterize the reliability of whether a target object exists in the target detection frame;
[0052] The local attention processing module is used to generate a local attention map corresponding to each monitoring area based on the image to be processed and each initial confidence level;
[0053] The global attention processing module is used to generate an average attention value of the region corresponding to the target detection frame in the global attention map according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map;
[0054] The confidence updating module is used to generate, for each target detection frame, an updated confidence corresponding to the target detection frame according to the target degree value, the regional average attention value, and the local attention map corresponding to the target detection frame; wherein the target degree value is used to characterize the degree of association between the target detection frame and other target detection frames in space and time;
[0055] The abnormality warning module is used to, when determining that an updated confidence is greater than a preset confidence threshold, use the corresponding target detection frame as a target frame to be warned, and generate an abnormality warning result corresponding to the target frame to be warned;
[0056] The training of the target detection model includes:
[0057] With several image samples as input and the actual target detection frame of each image sample and the corresponding actual confidence as output, the target detection model to be trained is iteratively trained until the model converges to generate a preset target detection model.
[0058] Based on the above method embodiments, the present invention provides corresponding terminal device embodiments.
[0059] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the image processing method based on a distributed monitoring system described in the above-mentioned embodiment of the invention.
[0060] The following beneficial effects are achieved by implementing the present invention:
[0061] The embodiment of the present invention provides an image processing method, apparatus and terminal device based on a distributed monitoring system. The present invention can extract a number of target detection frames that may contain target objects and their initial confidences from the to-be-processed images of each monitoring area through a preset target detection model, and can only focus on those local key areas that may contain target objects, thereby reducing the scanning and extraction in non-key areas (such as background areas), thereby realizing local scanning and local feature extraction of the monitoring area of the node camera. Furthermore, by generating the local attention map corresponding to each monitoring area and the regional average attention value corresponding to the global attention map, and updating the confidence of each target detection frame in combination with the target degree value, it is possible to focus more accurately on the key areas in the image, further reducing the waste of computing resources and improving image processing efficiency. By generating local attention maps, those areas that are more likely to contain target objects can be highlighted, and the global attention map can integrate the information of each local attention map, so that the attention distribution in the monitoring scene can be grasped as a whole, avoiding excessive attention or excessive neglect of a single local area, and by considering the spatial correlation degree of the target detection frame, the system can identify those target detection frames that are close to each other in position or have a specific spatial relationship, and the consideration of the temporal correlation degree can also enable the present invention to track the change trajectory of the target detection frame in continuous time, so that the real movement and noise interference (such as light and shadow changes, swaying leaves, etc.) of the target object in the target detection frame can be further accurately distinguished, so that the subsequent abnormal false alarm rate can be reduced. Compared with the prior art, the present invention can construct a multi-level attention mechanism from local to global by combining the spatial and temporal correlation degree of the local attention map, the global attention map and the target detection frame, which can accurately locate and identify specific targets in key areas of the local scene (such as the target object in the target detection frame), while reducing the waste of computing resources and the occurrence of false alarm rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 It is a flowchart of an image processing method based on a distributed monitoring system provided by an embodiment of the present invention.
[0063] Figure 2It is a structural diagram of the electric power Hongmeng operating system based on distributed camera provided by one embodiment of the present invention.
[0064] Figure 3 It is a structural diagram of an image processing device based on a distributed monitoring system provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] like Figure 1 FIG. 1 is a flow chart of an image processing method based on a distributed monitoring system provided by an embodiment of the present invention. The image processing method based on a distributed monitoring system includes:
[0067] Step S1: Acquire the image to be processed corresponding to each monitoring area in real time;
[0068] Step S2: For each image to be processed, the image to be processed is input into a preset target detection model, so that the target detection model extracts a number of target detection frames from the image to be processed, and generates an initial confidence level corresponding to each target detection frame; wherein the confidence level is used to characterize the reliability of whether the target object exists in the target detection frame; wherein the training of the target detection model includes:
[0069] Taking a number of image samples as input and the actual target detection frame of each image sample and the corresponding actual confidence as output, the target detection model to be trained is iteratively trained until the model converges to generate a preset target detection model;
[0070] Step S3: Based on the image to be processed and each initial confidence level, generate a local attention map corresponding to each monitoring area;
[0071] Step S4: Generate an average attention value of the region corresponding to the target detection box in the global attention map according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map;
[0072] Step S5: for each target detection frame, generate an updated confidence corresponding to the target detection frame according to the target degree value, the regional average attention value, and the local attention map corresponding to the target detection frame; wherein the target degree value is used to characterize the degree of association between the target detection frame and other target detection frames in space and time;
[0073] Step S6: When it is determined that an updated confidence level is greater than a preset confidence threshold, the corresponding target detection frame is used as a target frame to be warned, and an abnormal warning result corresponding to the target frame to be warned is generated.
[0074] For step S1, the embodiment of the present invention can obtain the monitoring image corresponding to each camera in the distributed monitoring system, and use the monitoring image as the image to be processed. Schematically, the corresponding monitoring target or the target object corresponding to the target detection frame to be detected in the corresponding image to be processed in different monitoring scenes will not be different. For example, in scenes such as traffic monitoring and parking lot management, vehicles are important targets that need to be marked. By marking the vehicle, the movement trajectory and status of the vehicle can be identified, and then operations such as traffic flow statistics or illegal parking detection can be performed. In monitoring scenes where personnel need to be monitored, different personnel are often important monitoring objects, such as staff, visitors, intruders, etc. By marking important personnel in the scene, the movement trajectory of the personnel can be detected and tracked, and then it can be determined whether there is abnormal behavior.
[0075] For step S2, in a preferred embodiment, when marking the targets in the monitoring scene, the present invention can use the target detection model to automatically identify and locate the target objects in each image to be processed, and generate a target detection box (Bounding Box) to mark their position and range. At the same time, the target detection model can also generate a confidence score for each target detection box to characterize the reliability of the target object in the detection box. Therefore, the present invention can extract and mark the local features of the monitoring image of each local area, so as to complete the target object marking of the local image, and then perform subsequent global analysis.
[0076] In a preferred embodiment, the training process of the target detection model includes:
[0077] Taking a number of image samples as input and the actual target detection frame of each image sample and the corresponding actual confidence as output, the target detection model to be trained is iteratively trained until the model converges to generate a preset target detection model;
[0078] During each training, an image sample may be input into the target detection model, so that the target detection model generates a target detection box prediction result and a confidence prediction result corresponding to the image sample according to the image features of the image sample;
[0079] The target detection box prediction results and confidence prediction results are compared with the actual target detection box results and actual confidence results corresponding to the image samples, and then the network parameters of the target detection model are adjusted according to the comparison results.
[0080] The model can be trained using image samples with actual target detection boxes and confidence levels, so that it can learn how to accurately identify and locate target objects in images. During the training process, the model will continuously adjust its network parameters based on the comparison between the predicted results and the actual annotations, thereby gradually reducing the model's prediction error and improving its accuracy.
[0081] For step S3, in a preferred embodiment, after obtaining each image to be processed, as well as the target detection frame in the image to be processed and the initial confidence corresponding to each target detection frame, the embodiment of the present invention can complete the calculation of the local attention mechanism.
[0082] Specifically, the image to be processed is the image to be processed acquired at the current moment; then when generating a local attention map corresponding to each monitoring area based on the image to be processed and each initial confidence level, it specifically includes:
[0083] Obtain a preset background image corresponding to the monitoring area at the previous moment, then compare the preset background image with the image to be processed, and generate a motion detection map according to the comparison result; wherein the motion detection map includes a background and a target in motion; the preset background image is a static image corresponding to the monitoring area;
[0084] Generate a current attention map corresponding to the current moment according to the historical local attention map corresponding to the previous moment, the historical attention map corresponding to the previous moment, and the attenuation factor; wherein the attenuation factor is used to characterize the contribution of historical information at the current moment;
[0085] According to each target detection frame, the initial confidence of each target detection frame, the motion detection map and the current attention map, a local attention map corresponding to each monitoring area is generated.
[0086] Indicatively, the local attention map corresponding to each monitoring area can be calculated according to the following formula:
[0087] ;
[0088] ;
[0089] ;
[0090] ;
[0091] in, is the local attention map corresponding to each monitoring area, The target detection box detection result corresponding to each monitoring area, is the initial confidence of the kth target detection box, is the binary mask of the kth target detection box, represents the sigmoid activation function, n is the number of target detection boxes; is the motion detection graph, is the image to be processed at the current moment, is the preset background image corresponding to the previous moment; is the current attention map corresponding to the current moment, is the historical local attention map corresponding to the previous moment of the monitoring area, is the historical attention map corresponding to the previous moment, is the attenuation factor; , are different weight coefficients, and + + =1.
[0092] Specifically, the target detection model may be a YOLO Nano model, and a preliminary detection result may be obtained in the final output layer of the YOLO Nano model. , including the position and confidence of the target detection box, that is, ,in is the binary mask of the k-th detection box, is the confidence of the corresponding detection box, is the sigmoid activation function.
[0093] Furthermore, in order to obtain a more accurate local attention map, the motion detection map can be obtained by using background subtraction. , by calculating the difference between the current frame and the previous frame (or background model), we have ,in is the image to be processed corresponding to the current frame (i.e. the current moment), It is the preset background image corresponding to the previous frame (i.e. the previous moment);
[0094] Furthermore, in order to consider the attention distribution of the previous frames, the attenuation factor is introduced (generally between 0.8 and 0.95), generate a historical attention map ;and ,in is the local attention map of the previous frame, is the historical attention map of the previous frame;
[0095] Finally, the local attention map corresponding to each monitoring area can be generated according to each target detection frame, each initial confidence, motion detection map and current attention map ; ,in, , are different weight coefficients, and + + =1.
[0096] Indicatively, the weight coefficient is preset to a fixed value and can be continuously adjusted, then:
[0097] ;
[0098] Different from the attention mechanism of Transformer, the embodiment of the present invention adopts a combination of feature extraction and weighting to obtain the weight coefficient.
[0099] By comparing the current image to be processed with the preset background image and generating a motion detection map, the moving target in the monitoring area can be accurately identified to reduce false detection and missed detection and improve the accuracy of target detection. By introducing the historical local attention map and the historical attention map, and balancing the contribution of historical information and current information through the attenuation factor, the target can be continuously tracked and focused in continuous monitoring.
[0100] In a preferred embodiment, the embodiment of the present invention can achieve the fusion of information such as the target detection frame, initial confidence, motion detection map and current attention map through the attention mechanism fusion model, and then generate a local attention map. A neural network structure (such as a convolutional neural network CNN) can be used in combination with an attention mechanism (such as a local attention mechanism) to achieve the generation of a local attention map, thereby achieving attention to and feature extraction of local areas within the monitoring area.
[0101] For step S4, in a preferred embodiment, generating the average attention value of the region corresponding to the target detection box in the global attention map according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map includes:
[0102] Generate a first global attention map at the current moment according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map;
[0103] Generate a second global attention map after time smoothing operation according to the historical global attention map at the previous moment, the first global attention map at the current moment, and a preset time smoothing coefficient;
[0104] Performing convolution calculation on the second global attention map to generate a third global attention map after spatial smoothing operation;
[0105] According to the coordinate information corresponding to the target detection box in the global coordinate system, an average attention value of the area corresponding to the target detection box in the third global attention map is generated.
[0106] Specifically, based on the regional communication function of the distributed soft bus of the Dianhong system, the camera nodes of the regional convergence layer can establish regional communication with the Dianhong cameras of the local processing layer, thereby summarizing the local attention maps of all Dianhong cameras in the region. , to generate the global attention map G. Among them, , where The function refers to the coordinate information corresponding to the local attention map converted to the global coordinate system. It refers to a local attention weight corresponding to each camera node i in the system. The default value is 1 / n, where n is the total number of cameras.
[0107] In order to improve the stability of the first global attention map, the embodiment of the present invention can also introduce time-space smoothing, that is, combining the historical global attention map of the previous frame in time. , and obtain the second global attention map after the time smoothing operation , ,in is the preset time smoothing coefficient;
[0108] In space, the two-dimensional convolution Conv2D is used to calculate the spatial smoothing operation, then:
[0109] ;
[0110] Among them, kernel=3, stride=1, padding=1, so that the size of the feature map can be kept unchanged, and It refers to the third global attention map after spatial smoothing operation.
[0111] Furthermore, the average attention value of the region corresponding to the target detection frame in the third global attention map can be generated according to the coordinate information corresponding to the target detection frame in the global coordinate system. , then:
[0112] ;
[0113] Schematically, (x1, y1, x2, y2) are the diagonal coordinates in the global coordinate system when the target detection box is a rectangular detection box.
[0114] The regional average attention value refers to a value obtained by averaging the attention values of all pixels in a specific area (such as the area covered by the target detection box) in the global attention map, which can reflect the importance or significance of the area in the global attention map. By calculating the regional average attention value of the target detection box in the global attention map, the importance of the area can be evaluated more accurately, so that the background and the target can be distinguished more accurately, reducing false detection and missed detection. When calculating the regional average attention value, the integral map technology can be used to speed up the calculation process, that is, the cumulative sum of all pixels above and to the left of each pixel in the image can be pre-calculated and stored, so as to quickly calculate the sum of any rectangular area.
[0115] For step S5, in a preferred embodiment, the target degree value used to characterize the degree of association between the target detection frame and other target detection frames in space and time includes: a first degree value and a second degree value;
[0116] The first degree value is used to represent the consistency of the detection of the target detection frame between different monitoring areas;
[0117] The second degree value is used to indicate whether the detection results corresponding to the target detection frame at different times maintain the same degree of stability;
[0118] Then, generating an updated confidence level corresponding to the target detection frame according to the target degree value, the average regional attention value, and the local attention map corresponding to the target detection frame specifically includes:
[0119] A monitoring area different from the monitoring area of the target detection frame is used as the target monitoring area, and then a first degree value is generated according to a determination result of whether there is a target detection frame identical to the target detection frame in the target monitoring area and an area weight value corresponding to each target monitoring area and used to characterize the size of the coverage area between two adjacent monitoring areas;
[0120] The first degree value can be calculated according to the following formula:
[0121] ;
[0122] in, is the first degree value, is the regional weight value, is the target detection frame detection result corresponding to a target monitoring area, is the number of target monitoring areas.
[0123] Further, according to a preset time interval, a number of historical moments are extracted; wherein each historical moment corresponds to a time weight value, and the smaller the time difference between a historical moment and the current moment, the larger the corresponding time weight value;
[0124] A second degree value is generated according to the target detection frame detection results corresponding to each historical moment and each time weight value; wherein the target detection frame detection result includes: detecting the target detection frame or not detecting the target detection frame; the second degree value can be calculated according to the following formula:
[0125] ;
[0126] in, is the second degree value, is the attenuation factor; is the time difference between the historical moment and the current moment, is the time weight value corresponding to each historical moment, Indicates the target detection frame detection result corresponding to the historical moment, =1 means the target detection frame is detected. =0 means that the target detection frame is not detected. is the number of historical moments;
[0127] Furthermore, an updated confidence level corresponding to the target detection frame may be generated based on the first degree value, the second degree value, the regional average attention value, and the local attention map corresponding to the target detection frame.
[0128] Specifically, the updated confidence can be calculated according to the following formula: :
[0129] ;
[0130] In the formula, The target detection box detection result corresponding to each monitoring area, is the average attention value of the region, is the first degree value, is the second degree value; , , , are different weight coefficients.
[0131] It is understandable that the first degree value It can be used to measure the consistency of detection results of the same target by different cameras. ,in is the detection result of the detection indicator of the jth camera (camera j is the other node except camera node i (i.e. the camera corresponding to the current target detection frame)), =1 means non-background target is detected, =0 means no target is detected, is the regional weight value, which represents the correlation between the i-th camera and the j-th camera and is positively correlated with the scene coverage area of the two cameras. In principle, the more different cameras detect the same target, The higher the value of , that is, different cameras are usually located in different positions and have different viewing angles. If multiple cameras can detect the same target, it means that the target is visible from multiple viewing angles, thereby increasing the possibility of its real existence. This multi-view verification is more reliable than the detection result of a single viewing angle, because through the introduction of the first degree value, the calculation and processing of the credibility of the real existence of the target can be increased, thereby obtaining a more accurate detection result.
[0132] Second degree value It can be used to measure the temporal stability of target detection results. The second degree value can be calculated according to the following formula: :
[0133] ;
[0134] in, is the second degree value, is the attenuation factor; is the time difference between the historical moment and the current moment, is the time weight value corresponding to each historical moment, Indicates the target detection frame detection result corresponding to the historical moment, =1 means the target detection frame is detected. =0 means that the target detection frame is not detected. is the number of historical moments.
[0135] When calculating the second degree value, each historical moment corresponds to a time weight value, and this weight value is usually inversely proportional to the time difference between the current moment and the current moment. In other words, the closer the historical frame is to the current moment, the greater its weight value. Therefore, when it is closer to the current moment, the more historical frames that can detect the target, and the frames with high weight values will make a greater contribution to the second degree value, making the second degree value higher.
[0136] Indicatively, a detection history queue can be generated and maintained with a length of 10 frames, and a time weight is assigned to each frame. , the closer the historical frame is to the current frame, The larger the value, Δt is the time interval between the last detection and the current moment. is the detection result corresponding to the historical moment t, =1 means the target detection frame is detected. =0 means that the target detection frame is not detected, and the more historical frames that can detect the target, the higher the second degree value It can be understood that if the target is stably detected in multiple consecutive frames, it means that the target is detected to exist continuously in time, rather than short-term noise or false alarm. Therefore, by considering the stability in time, the credibility of the real existence of the target can be considered.
[0137] In single-frame detection, random errors may occur due to various reasons (such as illumination changes, occlusion, noise, etc.), resulting in missed detection or false detection. When the detection results are consistent in multiple consecutive frames, the impact of these random errors will be averaged or offset, thereby improving the reliability of the detection results by calculating whether the detection results corresponding to the target detection frame at different times maintain the same degree of stability.
[0138] Therefore, the first degree value can reduce false detection or missed detection caused by limited viewing angle or error in a single monitoring area by comparing the detection results of the same target detection frame between different monitoring areas, thus achieving cross-region consistency verification and enhancing the accuracy of the detection results. The second degree value evaluates the stability of the target detection frame at different time points by considering the detection results and time weights of historical moments, so that it can identify and filter out those noises that appear or disappear briefly, and improve the detection credibility of long-term targets.
[0139] The present invention combines information of multiple dimensions (detection results in different monitoring areas and at different time points, and various attention maps). Even if there are errors or deficiencies in the information of a single dimension, information of other dimensions can provide effective supplement and verification.
[0140] In schematic form, by introducing the regional average attention value and the local attention map, effective allocation of computing resources can be achieved. For the areas with higher attention in the attention map, more computing resources and more sophisticated detection algorithms can be given to improve detection accuracy; while for the areas with lower attention, computing resources can be appropriately reduced to improve overall detection efficiency.
[0141] The present invention is not only suitable for simple monitoring scenarios, but can also maintain high detection performance in complex scenarios such as multi-camera coverage, complex background changes, and target occlusion, thus having high application value for applications in smart cities, security monitoring, autonomous driving and other fields.
[0142] For step S6, in a preferred embodiment, the image processing method of the present invention can be applied to a power Hongmeng operating system, such as Figure 2 As shown, it includes multiple electric cameras, and the multiple cameras are distributed in different monitoring positions, so that the system can collect image information at multiple locations at the same time, forming a multi-node distributed system.
[0143] In the electric Hongmeng operating system, the bottom-level camera belongs to the local processing layer, the middle-level camera is the regional aggregation layer, and the top video surveillance platform is the global analysis layer, which can form a multi-level processing architecture: local processing -> regional aggregation -> global analysis. It is understandable that the cameras communicate and interconnect through a distributed soft bus, and support dynamic role switching (ordinary node / regional aggregation node). The three-level processing architecture of the local processing layer, regional aggregation layer, and global analysis layer not only supports the vertical flow of data (from local to global), but also embodies the idea of distributed processing, that is, performing different processing tasks at different levels to optimize overall performance and efficiency.
[0144] It is understandable that at the local processing layer, the Dianhong camera device can perform image noise reduction and contrast enhancement on the collected images, complete preliminary image preprocessing, and run a lightweight target detection algorithm at the same time to identify potential areas of interest through the target detection model. At the regional convergence layer, it is responsible for integrating images and data from multiple local node cameras, realizing cross-camera target tracking and behavior analysis, and completing more complex analysis tasks. Finally, at the global analysis layer, the intelligent video surveillance platform can integrate data from all regions to conduct scene understanding and decision analysis at the intersection of local and global levels.
[0145] In a preferred embodiment, the present invention can not only perform local analysis and global analysis on each monitoring image to obtain a number of target detection boxes and confidence levels, but also implement abnormal alarms based on the obtained confidence levels.
[0146] First, a preset confidence threshold can be set, and this threshold can be determined according to the needs of the actual application scenario and the performance of the detection algorithm. When the updated confidence of a target detection box exceeds this threshold, it will be marked as a target box to be warned, indicating that the target object in the target box to be warned is considered abnormal or noteworthy in the current monitoring scenario.
[0147] For each marked target box to be warned, the system will also generate a corresponding abnormal alarm result. The alarm result may include information such as the location, size, type (such as people, cars, objects, etc.) and timestamp of the target. In a preferred embodiment, the alarm result may also include the alarm level or recommended countermeasures. The generated abnormal alarm result can be output in a variety of ways, such as displaying on the monitoring screen, sending text messages or emails to relevant personnel, triggering an alarm, or automatically executing preset countermeasures.
[0148] This application can monitor abnormal targets in the monitoring scene by automatically identifying the target detection frame and the updated confidence level, and can discover potential security risks, such as intruders, abnormal behaviors, etc., at the first time, so as to take timely measures to prevent the situation from escalating and ensure the safety of people and property.
[0149] In summary, the present invention combines local analysis with global analysis to obtain detection boxes and confidence levels, and implements abnormal alarms based on updated confidence levels, which not only improves the accuracy and efficiency of monitoring, but also enhances the security and decision-making support capabilities of the system.
[0150] like Figure 3 As shown, based on the above-mentioned various embodiments of the image processing method based on the distributed monitoring system, the present invention provides a corresponding device embodiment;
[0151] An embodiment of the present invention provides an image processing device based on a distributed monitoring system, comprising: an image acquisition module, an image processing module, a local attention processing module, a global attention processing module, a confidence update module, and an abnormal alarm module;
[0152] The image acquisition module is used to acquire the image to be processed corresponding to each monitoring area in real time;
[0153] The image processing module is used to input each image to be processed into a preset target detection model, so that the target detection model extracts a plurality of target detection frames from the image to be processed and generates an initial confidence level corresponding to each target detection frame; wherein the confidence level is used to characterize the reliability of whether a target object exists in the target detection frame;
[0154] The local attention processing module is used to generate a local attention map corresponding to each monitoring area based on the image to be processed and each initial confidence level;
[0155] The global attention processing module is used to generate an average attention value of the region corresponding to the target detection frame in the global attention map according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map;
[0156] The confidence updating module is used to generate, for each target detection frame, an updated confidence corresponding to the target detection frame according to the target degree value, the regional average attention value, and the local attention map corresponding to the target detection frame; wherein the target degree value is used to characterize the degree of association between the target detection frame and other target detection frames in space and time;
[0157] The abnormality warning module is used to, when determining that an updated confidence is greater than a preset confidence threshold, use the corresponding target detection frame as a target frame to be warned, and generate an abnormality warning result corresponding to the target frame to be warned;
[0158] The training of the target detection model includes:
[0159] With several image samples as input and the actual target detection frame of each image sample and the corresponding actual confidence as output, the target detection model to be trained is iteratively trained until the model converges to generate a preset target detection model.
[0160] It should be noted that the device embodiments described above are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without paying creative labor.
[0161] Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0162] On the basis of the above-mentioned various embodiments of the image processing method based on the distributed monitoring system, the present invention provides corresponding embodiments of terminal equipment items.
[0163] An embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements an image processing method based on a distributed monitoring system as described in any method embodiment of the present invention.
[0164] The terminal device may be a computing terminal device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.
[0165] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.
[0166] The memory can be used to store the computer program, and the processor realizes various functions of the terminal device by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device or other volatile solid-state storage device.
[0167] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. An image processing method based on a distributed monitoring system, characterized in that: include: Acquire the image to be processed corresponding to each monitoring area in real time; For each image to be processed, the image to be processed is input into a preset target detection model, so that the target detection model extracts a number of target detection frames from the image to be processed, and generates an initial confidence level corresponding to each target detection frame; wherein the confidence level is used to characterize the reliability of whether a target object exists in the target detection frame; Obtain a preset background image corresponding to the monitoring area at the previous moment, then compare the preset background image with the image to be processed, and generate a motion detection map according to the comparison result; wherein the motion detection map includes a background and a target in motion; the preset background image is a static image corresponding to the monitoring area; Generate a current attention map corresponding to the current moment according to the historical local attention map corresponding to the previous moment, the historical attention map corresponding to the previous moment, and the attenuation factor; wherein the attenuation factor is used to characterize the contribution of historical information at the current moment; Generate a local attention map corresponding to each monitoring area according to each target detection frame, the initial confidence of each target detection frame, the motion detection map and the current attention map; Generate a first global attention map at the current moment according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map; Generate a second global attention map after time smoothing operation according to the historical global attention map at the previous moment, the first global attention map at the current moment, and a preset time smoothing coefficient; Performing convolution calculation on the second global attention map to generate a third global attention map after spatial smoothing operation; Generate an average attention value of the region corresponding to the target detection frame in the third global attention map according to the coordinate information corresponding to the target detection frame in the global coordinate system; For each target detection frame, generate an updated confidence score corresponding to the target detection frame according to the target degree value, the regional average attention value, and the local attention map corresponding to the target detection frame; wherein the target degree value is used to characterize the degree of spatial and temporal association between the target detection frame and other target detection frames; When it is determined that an updated confidence is greater than a preset confidence threshold, the corresponding target detection frame is used as a target frame to be warned, and an abnormal alarm result corresponding to the target frame to be warned is generated; The training of the target detection model includes: Taking a number of image samples as input and the actual target detection frame of each image sample and the corresponding actual confidence as output, the target detection model to be trained is iteratively trained until the model converges to generate a preset target detection model; Among them, the local attention map corresponding to each monitoring area is calculated according to the following formula: ; ; ; ; in, is the local attention map corresponding to each monitoring area, The target detection box detection result corresponding to each monitoring area, is the initial confidence of the kth target detection box, is the binary mask of the kth target detection box, represents the sigmoid activation function, n is the number of target detection boxes; is the motion detection graph, is the image to be processed at the current moment, is the preset background image corresponding to the previous moment; is the current attention map corresponding to the current moment, is the historical local attention map corresponding to the previous moment of the monitoring area, is the historical attention map corresponding to the previous moment, is the attenuation factor; , are different weight coefficients, and + + =1.
2. The image processing method based on a distributed monitoring system according to claim 1, characterized in that: The target degree value includes: a first degree value and a second degree value; The generating, according to the target degree value, the regional average attention value and the local attention map corresponding to the target detection frame, an updated confidence corresponding to the target detection frame comprises: A monitoring area different from the monitoring area of the target detection frame is used as the target monitoring area, and then a first degree value is generated according to a determination result of whether there is a target detection frame identical to the target detection frame in the target monitoring area and a region weight value corresponding to each target monitoring area and used to characterize the size of the coverage area between two adjacent monitoring areas; wherein the first degree value is used to characterize the consistency degree of whether the target detection frame is detected consistently between different monitoring areas; Extracting a number of historical moments according to a preset time interval; wherein each historical moment corresponds to a time weight value, and the smaller the time difference between a historical moment and the current moment, the larger the corresponding time weight value; Generate a second degree value according to the target detection frame detection results corresponding to each historical moment and each time weight value; wherein the target detection frame detection result includes: detecting the target detection frame or not detecting the target detection frame; the second degree value is used to characterize whether the detection results corresponding to the target detection frame at different moments maintain the same degree of stability; According to the first degree value, the second degree value, the regional average attention value and the local attention map corresponding to the target detection frame, an updated confidence corresponding to the target detection frame is generated.
3. The image processing method based on a distributed monitoring system as claimed in claim 2, characterized in that: The generation of the first degree value includes: The first degree value is calculated according to the following formula: ; in, is the first degree value, is the regional weight value, is the target detection frame detection result corresponding to a target monitoring area, is the number of target monitoring areas.
4. The image processing method based on a distributed monitoring system as claimed in claim 3, characterized in that: The generating a second degree value according to the target detection frame detection results corresponding to each historical moment and each time weight value includes: The second degree value is calculated according to the following formula: ; in, is the second degree value, is the attenuation factor; is the time difference between a target historical moment and the current moment, wherein the time difference between the target historical moment and the current moment is the smallest; is the time weight value corresponding to each historical moment, Indicates the target detection frame detection result corresponding to the historical moment, =1 means the target detection frame is detected. =0 means that the target detection frame is not detected. is the number of historical moments.
5. The image processing method based on a distributed monitoring system according to claim 1, characterized in that: When iteratively training the target detection model to be trained, it also includes: During each training, an image sample is input into the target detection model so that the target detection model generates a target detection box prediction result and a confidence prediction result corresponding to the image sample according to the image features of the image sample; The target detection box prediction results and confidence prediction results are compared with the actual target detection box results and actual confidence results corresponding to the image samples, and then the network parameters of the target detection model are adjusted according to the comparison results.
6. An image processing device based on a distributed monitoring system, characterized in that: include: Image acquisition module, image processing module, local attention processing module, global attention processing module, confidence update module and abnormal alarm module; The image acquisition module is used to acquire the image to be processed corresponding to each monitoring area in real time; The image processing module is used to input each image to be processed into a preset target detection model, so that the target detection model extracts a plurality of target detection frames from the image to be processed and generates an initial confidence level corresponding to each target detection frame; wherein the confidence level is used to characterize the reliability of whether a target object exists in the target detection frame; The local attention processing module is used to obtain a preset background image corresponding to the monitoring area at the previous moment, and then compare the preset background image with the image to be processed, and generate a motion detection map according to the comparison result; wherein the motion detection map includes a background and a target in motion; the preset background image is a static image corresponding to the monitoring area; Generate a current attention map corresponding to the current moment according to the historical local attention map corresponding to the previous moment, the historical attention map corresponding to the previous moment, and the attenuation factor; wherein the attenuation factor is used to characterize the contribution of historical information at the current moment; Generate a local attention map corresponding to each monitoring area according to each target detection frame, the initial confidence of each target detection frame, the motion detection map and the current attention map; The global attention processing module is used to generate a first global attention map at a current moment according to the coordinate information corresponding to each local attention map in the global coordinate system and the local attention weight corresponding to each local attention map; Generate a second global attention map after time smoothing operation according to the historical global attention map at the previous moment, the first global attention map at the current moment, and a preset time smoothing coefficient; Performing convolution calculation on the second global attention map to generate a third global attention map after spatial smoothing operation; Generate an average attention value of the region corresponding to the target detection frame in the third global attention map according to the coordinate information corresponding to the target detection frame in the global coordinate system; The confidence updating module is used to generate, for each target detection frame, an updated confidence corresponding to the target detection frame according to the target degree value, the regional average attention value, and the local attention map corresponding to the target detection frame; wherein the target degree value is used to characterize the degree of association between the target detection frame and other target detection frames in space and time; The abnormality warning module is used to, when determining that an updated confidence is greater than a preset confidence threshold, use the corresponding target detection frame as a target frame to be warned, and generate an abnormality warning result corresponding to the target frame to be warned; The training of the target detection model includes: Taking a number of image samples as input and the actual target detection frame of each image sample and the corresponding actual confidence as output, the target detection model to be trained is iteratively trained until the model converges to generate a preset target detection model; Among them, the local attention map corresponding to each monitoring area is calculated according to the following formula: ; ; ; ; in, is the local attention map corresponding to each monitoring area, The target detection box detection result corresponding to each monitoring area, is the initial confidence of the kth target detection box, is the binary mask of the kth target detection box, represents the sigmoid activation function, n is the number of target detection boxes; is the motion detection graph, is the image to be processed at the current moment, is the preset background image corresponding to the previous moment; is the current attention map corresponding to the current moment, is the historical local attention map corresponding to the previous moment of the monitoring area, is the historical attention map corresponding to the previous moment, is the attenuation factor; , are different weight coefficients, and + + =1.
7. A terminal device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements an image processing method based on a distributed monitoring system as described in any one of claims 1 to 5.
Citation Information
Patent Citations
High-altitude operation safety monitoring method and device and storage medium
CN117253136A
Personnel behavior intelligent monitoring and alarming method and system
CN118072388A
Data processing method, data training method, data identifying method and device, and storage medium
WO2021057810A1
