Effective data supervision method, system and electronic equipment based on digitalization

Abstract

The present application relates to the field of data processing technology, and in particular to an effective data supervision method, system and electronic device based on digitalization, the method comprising: obtaining a first video captured by a core camera; performing target detection on the first video using a target detection model to obtain a detection result; if the target type is a normal type, determining the key features corresponding to the target type according to the correspondence between the type and the key features; determining whether the key features exist in the image corresponding to the target; when the preset conditions are met, determining the target as a suspicious target, and performing behavior recognition on the suspicious target based on the first video using a behavior recognition model to determine whether there is an abnormal behavior event. The present application can realize accurate detection and recognition of targets in the video captured by the core camera; in particular, when the target type is normal, by determining its key features and checking whether these features exist in the target image, suspicious targets can be effectively screened out.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an effective data supervision method, system and electronic equipment based on digitalization. Background Art

[0002] With the rapid development of information technology, digital transformation has become an important strategy for enterprises to enhance their competitiveness, optimize operational efficiency, and meet customer needs. In the wave of digital transformation, enterprises have an increasing demand for efficient, accurate, and intelligent digital services. Especially in the fields of intelligent manufacturing, intelligent transportation, and intelligent security, the application of target detection algorithms is particularly important. For example, in the field of intelligent manufacturing, target detection algorithms can realize automatic identification and positioning of parts on the production line; in the field of intelligent transportation, target detection algorithms can help realize functions such as traffic monitoring and vehicle tracking; in the field of intelligent security, target detection algorithms can be applied to scenarios such as face recognition and behavior analysis.

[0003] When it is necessary to monitor the behavior of people, only certain people may be allowed to enter and exit a certain area. At this time, these people who are allowed to enter and exit are normal types of targets, and people other than these people are suspicious persons. Conventionally, the video captured by the camera device is obtained, and the target detection model is used to determine whether the person is a normal type of person. If the person is not a normal type of person, the behavior recognition model is used to analyze the behavior of the suspicious target to determine whether abnormal behavior occurs. However, due to the influence of various factors, such as lighting conditions, occlusion, viewing angle, and the generalization ability of the model itself, when performing target detection through the target detection model, valuable targets may be mistakenly identified as worthless targets, that is, suspicious types of targets are identified as normal types of targets, resulting in false alarms and a high false alarm rate.

[0004] Therefore, how to improve the accuracy of target recognition has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] The purpose of this application is to provide an effective data supervision method, system and electronic equipment based on digitalization, which can effectively screen out suspicious targets and reduce the false alarm rate.

[0006] First, it provides an effective data supervision method based on digitalization, including:

[0007] Get the first video captured by the core camera;

[0008] Performing target detection on the first video using the target detection model to obtain a detection result, where the detection result includes: a target type and an image corresponding to the target;

[0009] If the target type is a normal type, determining the key features corresponding to the target type according to the correspondence between the type and the key features;

[0010] Determining whether the key feature exists in the image corresponding to the target;

[0011] When the preset conditions are met, the target is determined to be a suspicious target, and based on the first video, a behavior recognition model is used to perform behavior recognition on the suspicious target to determine whether there is an abnormal behavior event;

[0012] The preset condition is met as follows: the target type includes an abnormal type, or the key feature does not exist in the image corresponding to the target.

[0013] By adopting the above technical solution, it is possible to accurately detect and identify targets in the video captured by the core camera; especially when the target type is normal, by determining its key features and checking whether these features exist in the target image, suspicious targets can be effectively screened out and the false alarm rate can be reduced; by using the behavior recognition model to identify the behavior of suspicious targets, abnormal behavior events can be discovered in a timely manner, thereby improving the efficiency and accuracy of data supervision.

[0014] In a possible implementation, performing object detection on the first video using the object detection model to obtain a detection result includes:

[0015] Performing object detection on the first video using the object detection model to obtain a plurality of sub-detection results in a time series;

[0016] If for a certain target, there are a continuous preset number of sub-detection results in which the target types are all normal types, then the target type for the certain target in the detection result is determined to be a normal type; otherwise, it is an abnormal type.

[0017] By adopting the above technical solution, time series considerations are introduced into the target detection process, and the detection results are determined based on the sub-detection results of multiple consecutive frames. This processing method can more accurately reflect the actual state of the target, avoid misjudgment due to single-frame noise or errors, and improve the accuracy of target detection.

[0018] In a possible implementation, the method further includes:

[0019] If the key feature does not exist in the target image, misidentification information is recorded, and the misidentification information includes: the number of misidentifications and the image of the misidentified target;

[0020] When the number of misidentifications reaches a preset number threshold, the target detection model is trained again using a set of stored images of misidentified suspicious targets.

[0021] By adopting the above technical solution, when there are no key features in the target image, the misidentification information will be recorded. When the number of misidentifications reaches the preset threshold, the image set of these misidentified suspicious targets will be used to train the target detection model again. This feedback mechanism enables the model to continuously learn and optimize, reduce the future misidentification rate, and improve the reliability and stability of data supervision.

[0022] In a possible implementation, after performing behavior recognition on a suspicious target based on the first video and determining whether there is an abnormal behavior event by using a behavior recognition model, the method further includes:

[0023] If an abnormal event occurs, the moving speed and the moving direction of the suspicious target are calculated by the position change of the suspicious target in the image of the adjacent time frame in the first video;

[0024] Predict the movement trajectory of suspicious targets based on their direction of travel and traffic layout;

[0025] According to the motion trajectory and the pre-deployed camera association map, determine the slave camera corresponding to the area that the suspicious target will pass through when moving along the motion trajectory;

[0026] A second video corresponding to the camera is obtained, and based on the second video, a behavior recognition model is used to identify the suspicious target so as to track and monitor the suspicious target.

[0027] By adopting the above technical solution, after discovering an abnormal event, the movement speed and direction of the suspicious target are determined, its movement trajectory is predicted, and the slave camera corresponding to the area where the suspicious target will pass is determined based on the pre-deployed camera association map. By obtaining the corresponding video from the slave camera and re-identifying it using the behavior recognition model, tracking and monitoring of the suspicious target is achieved; this all-round, multi-perspective monitoring method can ensure continuous tracking and monitoring of suspicious targets, and improve the comprehensiveness and effectiveness of data supervision.

[0028] In a possible implementation, the identifying the suspicious target by using a behavior recognition model according to the second video includes:

[0029] Determine, according to the distance between the slave camera and the core camera, a behavior recognition threshold of a behavior recognition model for performing behavior recognition on the second video;

[0030] Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video.

[0031] By adopting the above technical solution, when performing behavior recognition on the video corresponding to the slave camera, the behavior recognition threshold of the behavior recognition model will be determined according to the distance between the slave camera and the core camera; this processing method takes into account the difference in distance between different cameras, and can more accurately identify the behavior of suspicious targets, especially when the distance is far or the viewing angle is biased. By adjusting the behavior recognition threshold, the accuracy and reliability of behavior recognition can be improved.

[0032] In a possible implementation, determining a behavior recognition threshold of a behavior recognition model for performing behavior recognition on the second video according to the distance between the slave camera and the core camera includes:

[0033] Estimate the distance from the suspicious target to the camera based on the motion trajectory and the position of the camera.

[0034] Determine the proportion of the suspicious object in the second video to the entire screen;

[0035] Determine the adjustment parameter according to the distance between the slave camera and the core camera and the target data, wherein the target data is the distance from the suspicious target to the slave camera or the proportion of the suspicious target in the entire screen in the second video;

[0036] According to the adjustment parameter, the preset behavior recognition threshold is modified to obtain the behavior recognition threshold of the behavior recognition model for the second video; the preset behavior recognition threshold is the recognition threshold of the behavior recognition model for the first video;

[0037] Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video.

[0038] By adopting the above technical solution, when determining the behavior recognition threshold for behavior recognition in the second video, multiple factors are comprehensively considered, such as the distance between the slave camera and the core camera, the estimated distance from the suspicious target to the slave camera, and the proportion of the suspicious target in the second video to the entire screen; this multi-factor comprehensive consideration method can more accurately determine the behavior recognition threshold, thereby improving the accuracy and reliability of the recognition of suspicious target behavior in the second video; especially in the case of a long distance or a small target area, by adjusting the behavior recognition threshold, timely recognition and monitoring of suspicious target behavior can be ensured.

[0039] In a possible implementation, the behavior recognition model includes: a target video frame behavior recognition network, an optical flow frame behavior recognition network, an original video behavior recognition network, and a decision network;

[0040] The training process of the behavior recognition model includes:

[0041] Acquire a sample video, wherein the sample video includes a plurality of samples and a behavior type corresponding to each sample image, wherein the sample includes an original video, an optical flow frame extracted based on the original video, and a target video frame where a suspicious target is determined based on the original video;

[0042] Inputting the target video frame of the sample into the target video frame behavior recognition network to obtain a first recognition result;

[0043] Inputting the optical flow frame extracted based on the original video into the optical flow frame behavior recognition network to obtain a second recognition result;

[0044] Input the original video into the original video behavior recognition network to obtain a third recognition result;

[0045] Inputting the first recognition result, the second recognition result and the third recognition result into a decision network to obtain a recognition result;

[0046] A loss value is determined according to the recognition result and the behavior type, and the behavior recognition model is iteratively trained according to the loss value to obtain a final behavior recognition model.

[0047] By adopting the above technical solution, the behavior recognition model is designed to include multiple different recognition networks, namely the target video frame behavior recognition network, the optical flow frame behavior recognition network, the original video behavior recognition network and the decision network. The video data is analyzed from multiple angles and levels, and by inputting the recognition results from these three different sources into the decision network, a more accurate and comprehensive behavior recognition result is obtained, thereby improving the efficiency and accuracy of abnormal event detection.

[0048] In a possible implementation, each recognition network includes: a feature extraction layer, a feature fusion layer, and a classification layer. The target video frame of the sample is input into the target video frame behavior recognition network to obtain a first recognition result; the optical flow frame extracted based on the original video is input into the optical flow frame behavior recognition network to obtain a second recognition result; the original video is input into the original video behavior recognition network to obtain a third recognition result, including:

[0049] Input the target video frame of the sample into the feature extraction layer, first obtain multiple first shallow features by downsampling, then obtain the first deep feature by upsampling, and obtain the first feature by the first deep feature and the first first shallow feature through the feature fusion layer; input the first feature into the classification layer to obtain the first recognition result;

[0050] The optical flow frame extracted from the original video is input into the optical flow frame behavior recognition network, multiple second shallow features are first obtained by downsampling, and then the second deep features are obtained by upsampling, and the second deep features and the first second shallow features are passed through the feature fusion layer to obtain the initial second features, and the initial second features are fused with the first features to obtain the second features; the second features are input into the classification layer to obtain the second recognition result;

[0051] The original video is input into the original video behavior recognition network, and multiple third shallow features are first obtained by downsampling, and then the third deep feature is obtained by upsampling, and the third deep feature and the first third shallow feature are passed through the feature fusion layer to obtain the initial third feature, and the initial third feature is fused with the second feature to obtain the third feature; the third feature is input into the classification layer to obtain the third recognition result.

[0052] By adopting the above technical scheme, each recognition network includes a feature extraction layer, a feature fusion layer and a classification layer. This structure can effectively extract key features from the input data, and fuse features at different levels through the feature fusion layer to obtain more comprehensive and rich information. For the target video frame behavior recognition network, multiple first shallow features are first obtained by downsampling, and then the first deep feature is obtained by upsampling, and the deep feature is fused with the first shallow feature. This method can simultaneously utilize the information of shallow and deep features, thereby improving the recognition accuracy. Similarly, the optical flow frame behavior recognition network and the original video behavior recognition network also adopt similar feature extraction and fusion methods. Finally, the recognition results of the three networks are fused through the decision network to obtain more accurate and reliable behavior recognition results, so that the behavior recognition model has stronger robustness and adaptability, and can cope with various complex and changeable situations.

[0053] Secondly, an effective data supervision system based on digitalization is provided, including:

[0054] An acquisition module, used to acquire a first video captured by a core camera;

[0055] The target detection module is used to perform target detection on the first video using the target detection model to obtain a detection result, where the detection result includes: a target type and an image corresponding to the target;

[0056] A feature determination module is used to determine the key features corresponding to the target type according to the correspondence between the type and the key features if the target type is a normal type; and determine whether the key features exist in the image corresponding to the target;

[0057] A behavior recognition module is used to determine that a target is a suspicious target when a preset condition is met, and based on the first video, use a behavior recognition model to perform behavior recognition on the suspicious target to determine whether there is an abnormal behavior event;

[0058] The preset condition is met as follows: the target type includes an abnormal type, or the key feature does not exist in the image corresponding to the target.

[0059] In a third aspect, an electronic device is provided, including:

[0060] one or more processors;

[0061] Memory;

[0062] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and one or more programs are configured to: perform operations corresponding to the method shown in any possible implementation of the first aspect.

[0063] In a fourth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, operations corresponding to the method shown in any possible implementation manner in the first aspect are implemented.

[0064] In summary, the present application includes at least one of the following beneficial technical effects:

[0065] 1. It can accurately detect and identify targets in the video captured by the core camera; especially when the target type is normal, by determining its key features and checking whether these features exist in the target image, it can effectively screen out suspicious targets and reduce the false alarm rate; using the behavior recognition model to identify the behavior of suspicious targets, it can timely discover abnormal behavior events, thereby improving the efficiency and accuracy of data supervision;

[0066] 2. After discovering an abnormal event, determine the movement speed and direction of the suspicious target, predict its movement trajectory, and determine the slave camera corresponding to the area where the suspicious target will pass based on the pre-deployed camera association map. By obtaining the corresponding video from the slave camera and re-identifying it using the behavior recognition model, tracking and monitoring of the suspicious target is achieved; this all-round, multi-perspective monitoring method can ensure continuous tracking and monitoring of suspicious targets, and improve the comprehensiveness and effectiveness of data supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a schematic diagram of an application scenario of an effective data supervision method based on digitalization provided in an embodiment of the present application;

[0068] Figure 2This is a flow chart of an effective data supervision method based on digitalization provided in an embodiment of the present application;

[0069] Figure 3 It is a schematic diagram of a model training provided in an embodiment of the present application;

[0070] Figure 4 is a schematic diagram of feature extraction provided in an embodiment of the present application;

[0071] Figure 5 It is a structural diagram of an effective data supervision system based on digitalization provided in an embodiment of the present application.

[0072] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] This specific embodiment is only an explanation of the present application, and it is not a limitation of the present application. After reading this specification, those skilled in the art can make modifications to the present embodiment without creative contribution as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law. It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the data related to the object is involved in the embodiments of the present application, it needs to be obtained through the authorization and consent of the object, the authorization and consent of the relevant departments, and in accordance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0076] Please refer to Figure 1 , Figure 1 A scenario diagram of an application of an effective data supervision method based on digitization provided in an embodiment of the present application, which can be applied to a data supervision system. In some embodiments, the data supervision system includes an electronic device, a terminal device, and a camera, the camera is used to collect video information for the electronic device to execute an effective data supervision method based on digitization, and the terminal device can digitally display the results. Among them, the terminal device includes but is not limited to: mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0077] When performing effective data supervision based on digitalization, the core camera collects the first video and sends it to the electronic device;

[0078] The electronic device uses the target detection model to perform target detection on the first video and obtains the detection result, which includes: the target type and the image corresponding to the target; if the target type is a normal type, the key features corresponding to the target type are determined according to the correspondence between the type and the key features; whether the key features exist in the image corresponding to the target; when the preset conditions are met, the target is determined to be a suspicious target, and based on the first video, the behavior recognition model is used to perform behavior recognition on the suspicious target to determine whether there is an abnormal behavior event; the preset conditions are met: the target type includes an abnormal type, or the key features do not exist in the image corresponding to the target. In this way, after the normal situation is detected by the target detection model, verification is performed again to ensure that even if there is a situation where the target can be identified as a normal type, it will be identified and corrected, and behavior recognition will be performed.

[0079] The terminal device displays the relevant information of the abnormal event visually on the display screen;

[0080] Furthermore, after analyzing the first video to find the presence of an abnormal event, the electronic device calculates the movement speed and direction of the suspicious target through the position changes of the suspicious target in the images of adjacent time frames in the first video; predicts the movement trajectory of the suspicious target based on the direction of movement and traffic layout; determines the slave camera corresponding to the area that the suspicious target will pass through when moving along the movement trajectory based on the movement trajectory and a pre-deployed camera association map; sends a second video from the camera to the electronic device; and the electronic device uses a behavior recognition model to identify the suspicious target based on the second video, so as to track and monitor the suspicious target.

[0081] The present application embodiment provides an effective data supervision method based on digitalization, such as Figure 2 As shown, the method provided in the embodiment of the present application can be executed by an electronic device, and the electronic device is a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The terminal device and the electronic device can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. The method includes:

[0082] S101, obtaining a first video captured by a core camera;

[0083] Core cameras are cameras installed at important locations. They have high resolution and shooting quality and can clearly capture people's behavior.

[0084] In some embodiments, the server first detects whether the core camera is connected and in working state. If the camera is connected and working properly, the server sends a command to the camera to start its video acquisition function. After receiving the command, the camera starts shooting and transmits video data to the server in real time, and these data are the first video.

[0085] In some embodiments, the server can also set a scheduled task or trigger condition to start the video acquisition of the camera. For example, the server can be set to automatically start the camera for data collection during a specific time period every day (such as a time period without human monitoring, or a time period of an important period), or when the server detects that a person enters the monitoring area, the camera is automatically triggered to start working.

[0086] In some embodiments, in order to ensure the stability and continuity of data, the server can also configure a video data cache mechanism. When the camera collects video data, the camera will first cache the data locally or in the cloud to ensure that stable video data can be continuously obtained even in network fluctuations or other abnormal situations.

[0087] S102, performing target detection on the first video using a target detection model to obtain a detection result, where the detection result includes: a target type and an image corresponding to the target;

[0088] The target detection model refers to a model trained by machine learning or deep learning algorithms. The model can identify the identity of the human target from the input video and then determine the target type based on the identity, and give their location images, which can be represented by a rectangular box or other forms, so as to quickly locate the target in the original video. Among them, the target detection model can identify the target. Among them, the target types include: normal type and suspicious type. The normal type corresponds to the person who is allowed to enter and exit, and the suspicious type corresponds to the person who is not allowed to enter and exit. When it is a normal type, the normal type also corresponds to the identity of the person. For example, when Zhang San is included in the first video, and Zhang San is a person who is allowed to enter, the detection result is normal type-Zhang San.

[0089] Specifically, the server will extract images frame by frame from the first video according to a preset frame rate or time interval, and pass these images as input data to the target detection model. The model will analyze each frame of the image, identify the target therein, and output the corresponding detection result.

[0090] In some embodiments, when the real-time requirement is not high, the first video can be split into continuous image frames, and then these image frames are passed to the object detection model one by one. The model will analyze each frame of the image independently, and output the sub-detection results of all objects in the frame, and generate a detection result based on the sub-detection results of all objects.

[0091] In other embodiments, when the real-time requirements are high, a streaming method is used to quickly detect the target in the video and provide real-time feedback to the user. The server directly passes the video stream to the target detection model. The model analyzes each frame in the video stream in real time and obtains continuous sub-detection results, thereby generating a detection result.

[0092] It is understandable that, in addition to the above two implementations, other technologies or algorithms can also be used to implement the operation of the target detection model, such as using GPU acceleration, distributed computing, etc., to improve the efficiency and accuracy of target detection. This is not limited here.

[0093] S103, if the target type is a normal type, determining the key features corresponding to the target type according to the correspondence between the type and the key features;

[0094] Key features refer to key information that can accurately describe the target type and are used to distinguish different types of targets or different individuals of the same type. Key features may include the height, appearance, and voiceprint of the person corresponding to the target type.

[0095] When the detected target type is determined to be a normal type, the server will consult a pre-established database or mapping table of correspondences between types and key features. This correspondence is determined during the server configuration phase, and it clarifies the key features corresponding to each normal type. By querying this correspondence, the server can quickly and accurately determine the key features of the current target type, providing a basis for subsequent target analysis or behavior recognition. Specifically, if the target type is a normal type - Zhang San, at this time, the key features corresponding to the target type that can be determined are Zhang San's key features, which can be his biological features, such as height, appearance or voiceprint, etc., which are not limited in the embodiments of this application.

[0096] Furthermore, the correspondence between target types and key features can be dynamically acquired and updated through external data sources or API interfaces. For example, when a new target type appears, the server can automatically acquire the key features of the type from the external data source and update them into the correspondence database.

[0097] S104, determining whether there is a key feature in the image corresponding to the target;

[0098] The image area corresponding to the target is analyzed and compared in detail to determine whether the image contains key features that match the given target type.

[0099] One way to implement this is to use a feature extraction algorithm to extract feature vectors from the image corresponding to the target, and then compare these feature vectors with key features that match the target type. If the similarity of the feature vectors reaches a certain threshold, it is considered that the key features exist in the image.

[0100] S105, when the preset conditions are met, determining the target as a suspicious target, and based on the first video, using a behavior recognition model, performing behavior recognition on the suspicious target to determine whether there is an abnormal behavior event;

[0101] The preset conditions are met if: the target type includes an abnormal type, or there is no key feature in the image corresponding to the target.

[0102] In this step, the preset conditions include two aspects: one is that the target type includes an abnormal type, that is, when the detected target type belongs to the pre-defined abnormal type list; the other is that the key features do not exist in the image corresponding to the target, that is, when the server detects that the image of the target does not match the key features, it is also considered to meet the preset conditions. The behavior recognition model is a model trained based on deep learning or machine learning technology, which is used to analyze and identify the behavior patterns of targets in videos or images. It can determine whether the target exhibits abnormal behavior based on the target's posture, movement and other information. "Behavioral abnormality events" refer to the behavior of the target that is significantly different from the normal behavior pattern, which may indicate that the target is engaging in illegal or dangerous activities.

[0103] Specifically, when the server detects that a target meets the preset conditions, the target will be automatically marked as a "suspicious target". Subsequently, the server will switch to the behavior recognition mode and use the pre-trained behavior recognition model to conduct an in-depth analysis of the part of the first video related to the suspicious target to determine whether there is an abnormal behavior event. If there is an abnormal behavior event, the corresponding alarm or processing mechanism will be triggered.

[0104] In some embodiments, the behavior identification of suspicious targets can be achieved in a variety of ways:

[0105] One way to achieve this is to use existing behavior recognition model libraries, such as OpenPose, DeepLab, etc. These models have been trained and optimized with a large amount of data and have high recognition accuracy and robustness. First, the server needs to extract the part related to the suspicious target in the first video and input it into the behavior recognition model. Then, the model will output feature information such as the target's motion trajectory and posture changes. Next, the server will compare this information with the preset normal behavior pattern to determine whether there is any abnormal behavior event.

[0106] Another implementation method is to customize the behavior recognition model. This method requires collecting and annotating a large amount of training data according to specific application scenarios and requirements. Then, use deep learning technology (such as convolutional neural networks, recurrent neural networks, etc.) to build a model, and train the model to accurately identify the target's behavior pattern. During the testing phase, the server can input the part of the first video related to the suspicious target into the customized model to obtain the target's behavior characteristics and make abnormal judgments.

[0107] It can be seen that in the embodiments of the present application, accurate detection and identification of targets in the video captured by the core camera can be achieved; especially when the target type is normal, suspicious targets can be effectively screened out by determining its key features and checking whether these features exist in the target image; using the behavior recognition model to perform behavior recognition on suspicious targets can timely discover abnormal behavior events, thereby improving the efficiency and accuracy of data supervision.

[0108] A possible implementation method of an embodiment of the present application is to use a target detection model to perform target detection on a first video to obtain a detection result, including: using a target detection model to perform target detection on the first video to obtain multiple sub-detection results with a time series; if for a certain target, there are a continuous preset number of sub-detection results in which the target types are all normal types, then determining that the target type for the certain target in the detection result is a normal type; otherwise, it is an abnormal type.

[0109] The sub-detection result is the result obtained by the target detection model after performing target detection on each frame or each video segment when processing the video. The time series refers to the sequence of these sub-detection results arranged in the time order of the video.

[0110] Specifically, when the target detection model processes the first video, it will first identify the targets in each frame or segment of the video and determine the types of these targets. Then, the model will record the type, position, size and other information of each target as a sub-detection result. As the video plays, the model will continuously detect new frames or segments and add new sub-detection results to the result sequence. Finally, when the video processing is completed, the server will obtain a sequence containing all sub-detection results, which are arranged in chronological order to form a detection result sequence with a time series. For a specific target, if the target type is normal in a preset number of consecutive sub-detection results, the server can determine that the type of the target in the entire video is normal; otherwise, the server will regard the target as an abnormal type.

[0111] It can be seen that in the embodiment of the present application, time series considerations are introduced in the target detection process, and the detection results are determined based on the sub-detection results of multiple consecutive frames. This processing method can more accurately reflect the actual state of the target, avoid misjudgment due to single-frame noise or errors, and improve the accuracy of target detection.

[0112] Based on any of the above embodiments, the method further includes:

[0113] If the key feature does not exist in the target image, the misidentification information is recorded, and the misidentification information includes: the number of misidentifications and the image of the misidentified target;

[0114] When the number of misidentifications reaches a preset threshold, the target detection model is trained again using a collection of stored images of misidentified suspicious targets.

[0115] Misidentification information refers to the information recorded when the model incorrectly identifies the target due to some reasons (such as poor image quality, target occlusion, etc.) during the target detection process. The number of misidentifications is a statistic that records the number of times misidentifications occur, and is used to evaluate the performance and accuracy of the target detection model. The images of misidentified targets refer to the image samples of the targets that are incorrectly identified by the model. These samples have important reference value for the improvement and optimization of subsequent models.

[0116] As the number of misidentifications accumulates, when the number of misidentifications reaches the preset threshold, it means that the model has a large recognition error in a specific scenario or condition and needs further optimization and adjustment. At this time, the server will use the previously stored collection of images of misidentified suspicious targets as new training data to train the target detection model again to improve the accuracy and generalization ability of the model.

[0117] Furthermore, when the number of misidentifications reaches a preset threshold, the model performance is evaluated. If the model performance is lower than the preset standard, the server will immediately trigger the model retraining process. During the retraining process, the server can adjust and optimize the model in a targeted manner according to the type and frequency of misidentifications to improve the recognition accuracy of the model in specific scenarios.

[0118] It can be seen that in the embodiment of the present application, when there are no key features in the image of the target, the misidentification information will be recorded. When the number of misidentifications reaches a preset threshold, the image set of these misidentified suspicious targets will be used to train the target detection model again. This feedback mechanism enables the model to continuously learn and optimize, reduce the future misidentification rate, and improve the reliability and stability of data supervision.

[0119] Based on any of the above embodiments, based on the first video, using the behavior recognition model, the behavior recognition of the suspicious target is performed to determine whether there is an abnormal behavior event, and further includes: S1-S4 (not shown in the drawings), wherein:

[0120] S1. If an abnormal event occurs, the movement speed and the moving direction of the suspicious target are calculated by the position change of the suspicious target in the image of the adjacent time frame in the first video;

[0121] Suspicious targets are objects in the first video that are associated with the abnormal event and need to be further analyzed. Position change refers to the change in the spatial position of the suspicious target relative to its surroundings between different time frames in the video. Movement speed refers to the distance moved by the suspicious target per unit time, which is obtained by calculating the position change of the target between consecutive time frames. The forward direction refers to the main direction of movement of the suspicious target in the video frame sequence, which can be determined by analyzing the trend of the target position change.

[0122] The speed and direction of movement of the suspicious target are calculated by analyzing the images of adjacent time frames in the first video, extracting the location information of the suspicious target, and calculating based on this location information. Example: Assume that in a surveillance video of a company's access control, no one should enter the company after get off work hours. Zhang San appears in the video and enters the company, which is an abnormal event. Zhang San is the suspicious target in this video. By analyzing the position changes of the car in the image, the speed of the car and its direction are calculated.

[0123] In some embodiments, the optical flow method is used to calculate the motion vectors of pixels between adjacent frames, thereby obtaining the motion trajectory of the suspicious target; by analyzing the trajectory, the motion speed and forward direction of the target are calculated.

[0124] S2. Predict the movement trajectory of suspicious targets based on the forward direction and traffic layout;

[0125] Traffic layout refers to the configuration and distribution of traffic elements such as roads, traffic signs, intersections, pedestrian walkways, etc. in a region or area. It reflects the operation rules of traffic flow and the setting of traffic facilities, and is the basis of traffic planning and management.

[0126] The motion trajectory refers to the prediction of the path or route that a target may take over a period of time based on its direction and speed. In video surveillance, the prediction of motion trajectory helps to understand the target's action intentions, thereby conducting effective monitoring and early warning.

[0127] This step predicts the path or trajectory that the target may take in the future by analyzing the direction of the suspicious target in the video and combining it with the traffic layout of the area. Example: Suppose in a surveillance video of a company's access control, no one should enter the company after get off work hours. Zhang San appears in the video and enters the company. His direction of travel is clearly pointed to an office at the end of the passage. At this time, combined with the traffic layout of the intersection, it can be predicted that Zhang San may enter the office along the passage in the next few seconds. This is the prediction of the movement trajectory.

[0128] Optionally, one implementation method is to use a trajectory prediction method based on a physical model. Establish a physical model that describes the target's motion law, such as Newton's laws of motion or a Kalman filter. Then, based on the target's direction of travel and initial position, combined with traffic layout information (such as road width, curvature, etc.), the physical model is used to calculate the target's possible position in the future, thereby obtaining its motion trajectory. Another implementation method is to use a trajectory prediction method based on machine learning. This method requires first collecting a large amount of historical trajectory data and building a machine learning model for predicting trajectories. Then, the direction of travel and traffic layout information of the suspicious target are used as input and input into the model for training and prediction. The model learns the characteristics of the historical trajectory data and predicts the future motion trajectory of the suspicious target based on it.

[0129] S3, according to the motion trajectory and the pre-deployed camera association map, determine the slave camera corresponding to the area that the suspicious target will pass through when moving along the motion trajectory;

[0130] The pre-deployed camera association map represents the camera network pre-planned and set up in the monitoring area and the associations between them; this association map may include information such as the camera's location, coverage, monitoring angle, resolution, and the spatial and temporal associations between cameras. The slave camera refers to the camera that can capture the area where the suspicious target passes when moving along the motion trajectory, determined based on the motion trajectory and the camera association map. These cameras can provide key video data during the target's motion, which is helpful for further analysis and tracking.

[0131] Example: Assume that in a shopping mall's surveillance server, a suspicious person's movement trajectory from the entrance to the cashier counter is determined through calculation. Based on the pre-deployed camera association map, multiple cameras that can cover the movement trajectory can be found, such as cameras at the entrance, cameras in the corridor, and cameras near the cashier counter.

[0132] In this step, the server has determined the movement trajectory of the suspicious target through analysis. Next, it is necessary to find a camera that can cover this movement trajectory in order to obtain more information about the suspicious target.

[0133] Specifically, the server will first extract the movement trajectory data of the suspicious target, including the target's starting position, end position, and key points on the path. Then, the server will compare and analyze these trajectory data with the pre-stored camera association map; the server will consider factors such as the camera's location, coverage, and monitoring angle to determine which cameras can cover each area on the target's movement trajectory. Finally, the server will output the information of these cameras as slave cameras for subsequent analysis and tracking.

[0134] S4. Obtain a second video corresponding to the camera, and identify the suspicious target using a behavior recognition model based on the second video to track and monitor the suspicious target.

[0135] It can be seen that in the embodiment of the present application, after an abnormal event is discovered, the movement speed and direction of the suspicious target are determined, its movement trajectory is predicted, and the slave camera corresponding to the area where the suspicious target will pass is determined according to the pre-deployed camera association map. Then, by obtaining the corresponding video from the slave camera and re-identifying it using the behavior recognition model, tracking and monitoring of the suspicious target is achieved; this all-round, multi-perspective monitoring method can ensure continuous tracking and monitoring of suspicious targets, and improve the comprehensiveness and effectiveness of data supervision.

[0136] Based on any of the above embodiments, according to the second video, the suspicious target is identified by using a behavior recognition model, including:

[0137] Determine, according to the distance between the slave camera and the core camera, a behavior recognition threshold of a behavior recognition model for performing behavior recognition on the second video;

[0138] Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video.

[0139] Slave cameras are other cameras used to assist monitoring and capture the target's movement trajectory in addition to the core camera; they are usually distributed in various corners of the monitoring area to provide additional viewing angles and coverage.

[0140] The behavior recognition threshold is a threshold used to determine whether the output result of the behavior recognition model is reliable. When the behavior recognition score output by the model is higher than this threshold, the server considers that the behavior is successfully recognized; otherwise, it considers that the recognition fails or the result is unreliable. In the embodiment of the present application, the behavior recognition model is a recognition model for multiple categories of behaviors.

[0141] Among them, in the video surveillance server, the server will dynamically adjust the behavior recognition threshold of the behavior recognition model when processing the second video captured by the slave camera according to the physical distance between the slave camera and the core camera. This is because the video quality, target size and other factors captured by cameras at different distances may be different, so different thresholds are required to ensure the accuracy of behavior recognition. Suppose in a monitoring server of a large square, the core camera is located in the center of the square, and the slave cameras are distributed around the square. When the slave camera is far away from the core camera, the target in the captured video may be small and blurred. At this time, the behavior recognition threshold needs to be lowered to improve the accuracy of recognition; when the slave camera is close to the core camera, the target in the captured video is large and clear. At this time, the behavior recognition threshold can be appropriately increased to reduce false alarms.

[0142] The server will pass this threshold as a parameter to the behavior recognition model so that the model can output reliable behavior recognition results based on this threshold when processing the second video. Then, when the server compares the behavior recognition results output by the model with the preset warning conditions, an alarm or other corresponding response will be triggered if the conditions are met.

[0143] It can be seen that in the embodiment of the present application, when performing behavior recognition on the video corresponding to the slave camera, the behavior recognition threshold of the behavior recognition model will be determined based on the distance between the slave camera and the core camera; this processing method takes into account the differences in distances between different cameras, and can more accurately identify the behavior of suspicious targets, especially when the distance is far or the viewing angle is biased. By adjusting the behavior recognition threshold, the accuracy and reliability of behavior recognition can be improved.

[0144] In some embodiments, a mapping table of distance and threshold is established, and the behavior recognition threshold is determined through the mapping table. The server can search for the corresponding threshold in the mapping table according to the actual distance between the slave camera and the core camera; this method is simple and direct, but it requires to determine the mapping relationship in advance through experiments or experience, or to determine the threshold corresponding to the distance based on the distance between the slave camera and the core camera, video quality, target size and other information during the test, as well as the corresponding behavior recognition results.

[0145] In some other embodiments, determining the behavior recognition threshold of the behavior recognition model for performing behavior recognition on the second video according to the distance between the slave camera and the core camera includes: SA1-SA5 (not shown in the drawings), wherein:

[0146] SA1. Estimate the distance from the suspicious target to the camera based on the motion trajectory and the position of the camera.

[0147] SA2, determining the proportion of the suspicious target in the second video to the entire screen;

[0148] The distance from the suspicious target to the camera indicates the straight-line distance between the suspicious target and the camera, calculated or estimated based on the movement trajectory of the suspicious target and the position of the camera. The proportion of the suspicious target in the entire screen reflects the prominence and clarity of the target in the video.

[0149] Optionally, estimating the distance from the suspicious target to the slave camera can be achieved by the following steps: determining the monitoring area that can be covered by the slave camera based on its position and field of view, and calculating the straight-line distance between the target and the slave camera based on the movement trajectory of the suspicious target. Furthermore, the calculated distance can be corrected by taking into account factors such as camera distortion and installation angle.

[0150] SA3. Determine adjustment parameters according to the distance between the slave camera and the core camera and target data, where the target data is the distance from the suspicious target to the slave camera or the proportion of the suspicious target in the entire screen in the second video;

[0151] The target data reflects the size and prominence of the suspicious target in the video footage captured by the camera.

[0152] The adjustment parameters are one or more parameter values ​​calculated based on the distance between the camera and the core camera and the target data, and these parameter values ​​are used to adjust certain settings in the video analysis or target recognition algorithm to adapt to different monitoring environments and target conditions.

[0153] Specifically, the server will first obtain the distance information between the slave camera and the core camera, as well as the target data in the video data captured by the camera. Then, the adjustment parameters are calculated based on these two factors. These adjustment parameters will be used to adjust the relevant parameters in the target recognition or behavior analysis algorithm to improve the accuracy and robustness of the algorithm. The adjustment parameters are used to lower the threshold so that more comprehensive valid information can be obtained from the video captured by the camera. The greater the distance, the smaller the threshold, and the smaller the area, the smaller the threshold.

[0154] One implementation method is to use a preset formula or algorithm to calculate the adjustment parameters. First, based on the distance between the slave camera and the core camera, and the target data (the distance from the suspicious target to the slave camera or the proportion of the suspicious target in the picture), the server can calculate some intermediate values, such as the actual size of the target, the relative position of the target, etc. Then, based on these intermediate values ​​and the preset formula or algorithm, the adjustment parameters are calculated. Finally, the adjustment parameters are applied to the target recognition or behavior analysis algorithm. The formula in this application can be y=ax+b, or y=ax, or y=x+b, where a and b are adjustment parameters.

[0155] Therefore, optionally, y=ax+b is selected, and parameter a is determined based on the distance between the slave camera and the core camera in combination with the first mapping relationship, where the first mapping relationship is the mapping relationship between the distance and parameter a; parameter b is then determined through the target data in combination with the second mapping relationship, where the second mapping relationship is the mapping relationship between the data and parameter b.

[0156] Optionally, select y=ax, and determine parameter a1 according to the distance between the slave camera and the core camera in combination with the third mapping relationship, where the third mapping relationship is the mapping relationship between the distance and parameter a1; then determine parameter a2 through the target data in combination with the fourth mapping relationship, where the fourth mapping relationship is the mapping relationship between the data and parameter a2; perform weighted calculation based on a1 and a2 to obtain a.

[0157] Optionally, select y=x+b, and determine parameter b1 based on the distance between the slave camera and the core camera in combination with the fifth mapping relationship, which is the mapping relationship between the distance and parameter b1; then determine parameter b2 through the target data in combination with the sixth mapping relationship, which is the mapping relationship between the data and parameter b2; perform weighted calculation based on b1 and b2 to obtain a.

[0158] SA4. According to the adjustment parameter, the preset behavior recognition threshold is modified to obtain the behavior recognition threshold of the behavior recognition model for the second video; the preset behavior recognition threshold is the recognition threshold of the behavior recognition model for the first video;

[0159] SA5. Based on the behavior recognition threshold, use the behavior recognition model to perform behavior recognition on the suspicious target in the second video.

[0160] It can be seen that in the embodiment of the present application, when determining the behavior recognition threshold for behavior recognition of the second video, multiple factors are comprehensively considered, such as the distance between the slave camera and the core camera, the estimated distance from the suspicious target to the slave camera, and the proportion of the suspicious target in the second video to the entire screen; this multi-factor comprehensive consideration method can more accurately determine the behavior recognition threshold, thereby improving the accuracy and reliability of the recognition of suspicious target behavior in the second video; especially when the distance is far or the target area is small, by adjusting the behavior recognition threshold, timely identification and monitoring of suspicious target behavior can be ensured.

[0161] Based on any of the above embodiments, refer to Figure 3 ,The behavior recognition model includes : target video frame behavior recognition network, optical flow frame behavior recognition network, original video behavior recognition network and decision ,network;

[0162] The training process of the behavior recognition model includes:

[0163] Obtaining a sample video, the sample video including a plurality of samples and a behavior type corresponding to each sample image, the sample including an original video, an optical flow frame extracted based on the original video, and a target video frame containing a suspicious target determined based on the original video;

[0164] Inputting the target video frame of the sample into the target video frame behavior recognition network to obtain a first recognition result;

[0165] Inputting the optical flow frame extracted based on the original video into the optical flow frame behavior recognition network to obtain a second recognition result;

[0166] Input the original video into the original video behavior recognition network to obtain a third recognition result;

[0167] Inputting the first recognition result, the second recognition result and the third recognition result into a decision network to obtain a recognition result;

[0168] According to the recognition result and behavior type, the loss value is determined, and the behavior recognition model is iteratively trained according to the loss value to obtain the final behavior recognition model.

[0169] Raw video refers to video data that is directly shot without any processing, and it contains all the necessary visual information. Optical flow frames are extracted from raw video through a specific algorithm and are used to describe the image sequence of the movement of objects in the video. Optical flow technology can capture the changes in pixel intensity and movement trends in the video, thereby providing additional information about the movement of objects.

[0170] The target video frames are determined based on the original video through the target detection algorithm. These frames contain suspicious targets, that is, objects or areas that need to be identified and analyzed. The extraction of target video frames helps to focus on key areas and improve the accuracy of behavior recognition.

[0171] Specifically, during the training process, each sample video will be split into multiple samples. Each sample contains a piece of video data (original video, optical flow frame or target video frame) and the corresponding behavior type label. These samples are input into the corresponding behavior recognition network for training. Each network contains multiple levels and parameters. These parameters are optimized through continuous forward propagation and back propagation, so that the network can more accurately identify the behavior in the video. The recognition results of multiple networks are input into the decision network for comprehensive decision-making to obtain a more accurate and reliable recognition result.

[0172] It can be seen that in the embodiment of the present application, the behavior recognition model is designed to include multiple different recognition networks, namely, the target video frame behavior recognition network, the optical flow frame behavior recognition network, the original video behavior recognition network, and the decision network. Video data is analyzed from multiple angles and levels, and by inputting the recognition results from these three different sources into the decision network, a more accurate and comprehensive behavior recognition result is obtained, thereby improving the detection efficiency and accuracy of abnormal events.

[0173] Specifically, refer to Figure 4 , each recognition network includes: a feature extraction layer, a feature fusion layer, and a classification layer. The target video frame of the sample is input into the target video frame behavior recognition network to obtain a first recognition result; the optical flow frame extracted based on the original video is input into the optical flow frame behavior recognition network to obtain a second recognition result; the original video is input into the original video behavior recognition network to obtain a third recognition result, including:

[0174] Input the target video frame of the sample into the feature extraction layer, first obtain multiple first shallow features by downsampling, then obtain the first deep feature by upsampling, and obtain the first feature by the first deep feature and the first first shallow feature through the feature fusion layer; input the first feature into the classification layer to obtain the first recognition result;

[0175] The optical flow frame extracted from the original video is input into the optical flow frame behavior recognition network, multiple second shallow features are first obtained by downsampling, and then the second deep features are obtained by upsampling, and the second deep features and the first second shallow features are passed through the feature fusion layer to obtain the initial second features, and the initial second features are fused with the first features to obtain the second features; the second features are input into the classification layer to obtain the second recognition result;

[0176] The original video is input into the original video behavior recognition network, and multiple third shallow features are first obtained by downsampling, and then the third deep feature is obtained by upsampling, and the third deep feature and the first third shallow feature are passed through the feature fusion layer to obtain the initial third feature, and the initial third feature is fused with the second feature to obtain the third feature; the third feature is input into the classification layer to obtain the third recognition result.

[0177] In an embodiment of the present application, the feature extraction layer is used to extract key information or features from the input data. Downsampling is used to reduce the resolution or size of the image while retaining its main information. In the feature extraction process, downsampling helps to reduce the amount of calculation and memory consumption while maintaining the capture of key information of the image. Upsampling is the inverse process of downsampling, which is used to increase the resolution or size of the image. In the feature extraction process, upsampling is usually used to restore some detail information of the image in order to better represent the high-level features of the image. Multiple first shallow features refer to features extracted by downsampling in the early stage of the feature extraction layer, which usually describe the local or detail information of the image. The first deep feature is a feature obtained after upsampling, which usually describes the global or high-level information of the image. The feature fusion layer is used to fuse features from different layers or different sources to obtain a richer feature representation. The classification layer is used to predict or classify categories based on the input features. The first recognition result, the second recognition result, and the third recognition result respectively represent the behavior recognition results based on the target video frame, the optical flow frame, and the original video.

[0178] Specifically, the target video frame of the sample is input into the feature extraction layer, and through downsampling and upsampling operations, as well as the feature fusion layer, the features that can describe the target behavior (i.e., the first feature) are gradually extracted and fused. The first feature is input into the classification layer to obtain the first recognition result based on the target video frame. The optical flow frame extracted based on the original video is input into the optical flow frame behavior recognition network, and a similar feature extraction and fusion process is performed to obtain the second feature. The second feature is then fused with the first feature to obtain a richer feature representation (i.e., the second feature), which is then input into the classification layer to obtain the second recognition result. The original video is input into the original video behavior recognition network, and feature extraction, fusion, and classification are performed again to obtain the third feature and the third recognition result.

[0179] Of course, in the embodiment of the present application, when downsampling, each layer can calculate the residual calculation twice to obtain the corresponding shallow feature map, and use the obtained shallow feature map as the input of the downsampling of the next layer until all shallow feature maps are obtained, and then the shallow feature map obtained by the last n layers is calculated with the feature map of the n-1 layer for attention calculation, and then the calculated features are fused with the shallow feature map obtained by the n layers, and the residual calculation is performed to obtain the deep features of the n-1 layer, so as to continue upsampling and fusion until the first feature is obtained. The specific process of the embodiment of the present application will not be repeated.

[0180] It can be seen that in the embodiment of the present application, each recognition network includes a feature extraction layer, a feature fusion layer and a classification layer. This structure can effectively extract key features from the input data, and fuse features at different levels through the feature fusion layer to obtain more comprehensive and rich information. For the target video frame behavior recognition network, multiple first shallow features are first obtained by downsampling, and then the first deep feature is obtained by upsampling, and the deep feature is fused with the first shallow feature. This method can simultaneously utilize the information of shallow and deep features, thereby improving the recognition accuracy. Similarly, the optical flow frame behavior recognition network and the original video behavior recognition network also adopt similar feature extraction and fusion methods. Finally, the recognition results of the three networks are fused through the decision network to obtain more accurate and reliable behavior recognition results, so that the behavior recognition model has stronger robustness and adaptability, and can cope with various complex and changeable situations.

[0181] An effective data supervision system based on digitization provided in an embodiment of the present application is introduced below. The effective data supervision system based on digitization described below and the effective data supervision method based on digitization described above can be referred to each other. The effective data supervision system based on digitization of this embodiment is set in an electronic device. Figure 5 , Figure 5 This is a structural block diagram of an effective data supervision system based on digitalization according to one embodiment of the present application, including:

[0182] An acquisition module 10 is used to acquire a first video captured by a core camera;

[0183] The target detection module 20 is used to perform target detection on the first video using the target detection model to obtain a detection result, where the detection result includes: a target type and an image corresponding to the target;

[0184] The feature determination module 30 is used to determine the key features corresponding to the target type according to the correspondence between the type and the key features if the target type is a normal type; and determine whether the key features exist in the image corresponding to the target;

[0185] The behavior recognition module 40 is used to determine that the target is a suspicious target when the preset conditions are met, and to perform behavior recognition on the suspicious target based on the first video using the behavior recognition model to determine whether there is an abnormal behavior event;

[0186] The preset conditions are met if: the target type includes an abnormal type, or there is no key feature in the image corresponding to the target.

[0187] In one achievable manner, the object detection module 20 performs object detection on the first video using the object detection model to obtain a detection result for:

[0188] Performing object detection on the first video using the object detection model to obtain a plurality of sub-detection results in a time series;

[0189] If, for a certain target, there are a continuous preset number of sub-detection results in which the target types are all normal types, then the target type for the certain target in the detection result is determined to be a normal type; otherwise, it is an abnormal type.

[0190] In one achievable manner, the present invention further includes:

[0191] A recording module, used for recording misidentification information if the key feature does not exist in the target image, the misidentification information including: the number of misidentifications and the image of the misidentified target;

[0192] The iterative training module is used to train the target detection model again by storing a set of images of misidentified suspicious targets when the number of misidentifications reaches a preset threshold.

[0193] In one achievable manner, the present invention further includes:

[0194] A calculation module, for calculating the movement speed and the forward direction of the suspicious target by the position change of the suspicious target in the image of the adjacent time frame in the first video if an abnormal event occurs;

[0195] Prediction module, used to predict the movement trajectory of suspicious targets based on the forward direction and traffic layout;

[0196] A slave camera determination module is used to determine the slave camera corresponding to the area that the suspicious target will pass through when moving along the motion trajectory according to the motion trajectory and the pre-deployed camera association map;

[0197] The tracking and monitoring module is used to obtain the second video corresponding to the camera, and use the behavior recognition model to identify the suspicious target based on the second video to track and monitor the suspicious target.

[0198] In an achievable manner, the tracking monitoring module uses a behavior recognition model to identify a suspicious target based on the second video, for:

[0199] Determine, according to the distance between the slave camera and the core camera, a behavior recognition threshold of a behavior recognition model for performing behavior recognition on the second video;

[0200] Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video.

[0201] In one achievable manner, the tracking monitoring module determines, according to the distance between the slave camera and the core camera, a behavior recognition threshold of the behavior recognition model for performing behavior recognition on the second video, for:

[0202] Estimate the distance from the suspicious target to the camera based on the motion trajectory and the position of the camera.

[0203] Determine the proportion of the suspicious object in the second video to the entire screen;

[0204] Determine the adjustment parameter according to the distance between the slave camera and the core camera and the target data, where the target data is the distance from the suspicious target to the slave camera or the proportion of the suspicious target in the entire screen in the second video;

[0205] According to the adjustment parameter, the preset behavior recognition threshold is modified to obtain the behavior recognition threshold of the behavior recognition model for the second video; the preset behavior recognition threshold is the recognition threshold of the behavior recognition model for the first video;

[0206] Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video.

[0207] In one achievable manner, the behavior recognition model includes: a target video frame behavior recognition network, an optical flow frame behavior recognition network, an original video behavior recognition network, and a decision network;

[0208] Also included: Training modules for:

[0209] Obtaining a sample video, the sample video including a plurality of samples and a behavior type corresponding to each sample image, the sample including an original video, an optical flow frame extracted based on the original video, and a target video frame containing a suspicious target determined based on the original video;

[0210] Inputting the target video frame of the sample into the target video frame behavior recognition network to obtain a first recognition result;

[0211] Inputting the optical flow frame extracted based on the original video into the optical flow frame behavior recognition network to obtain a second recognition result;

[0212] Input the original video into the original video behavior recognition network to obtain a third recognition result;

[0213] Inputting the first recognition result, the second recognition result and the third recognition result into a decision network to obtain a recognition result;

[0214] According to the recognition result and behavior type, the loss value is determined, and the behavior recognition model is iteratively trained according to the loss value to obtain the final behavior recognition model.

[0215] In an achievable manner, each recognition network includes: a feature extraction layer, a feature fusion layer, and a classification layer. The training module inputs the target video frame of the sample into the target video frame behavior recognition network to obtain a first recognition result; inputs the optical flow frame extracted based on the original video into the optical flow frame behavior recognition network to obtain a second recognition result; and inputs the original video into the original video behavior recognition network to obtain a third recognition result, which is used for:

[0216] Input the target video frame of the sample into the feature extraction layer, first obtain multiple first shallow features by downsampling, then obtain the first deep feature by upsampling, and obtain the first feature by the first deep feature and the first first shallow feature through the feature fusion layer; input the first feature into the classification layer to obtain the first recognition result;

[0217] The optical flow frame extracted from the original video is input into the optical flow frame behavior recognition network, multiple second shallow features are first obtained by downsampling, and then the second deep features are obtained by upsampling, and the second deep features and the first second shallow features are passed through the feature fusion layer to obtain the initial second features, and the initial second features are fused with the first features to obtain the second features; the second features are input into the classification layer to obtain the second recognition result;

[0218] The original video is input into the original video behavior recognition network, and multiple third shallow features are first obtained by downsampling, and then the third deep feature is obtained by upsampling, and the third deep feature and the first third shallow feature are passed through the feature fusion layer to obtain the initial third feature, and the initial third feature is fused with the second feature to obtain the third feature; the third feature is input into the classification layer to obtain the third recognition result.

[0219] An electronic device is provided in an embodiment of the present application, such as Figure 6 As shown, Figure 6 The electronic device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

[0220] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0221] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0222] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0223] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0224] Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0225] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0226] An embodiment of the present application provides a computer program product, including a computer program, which implements the corresponding contents of the aforementioned method embodiment when the computer program is executed by a processor.

[0227] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0228] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An effective data supervision method based on digitalization, characterized in that: include: Get the first video captured by the core camera; Performing target detection on the first video using the target detection model to obtain a detection result, where the detection result includes: a target type and an image corresponding to the target; If the target type is a normal type, determining the key features corresponding to the target type according to the correspondence between the type and the key features; Determining whether the key feature exists in the image corresponding to the target; When the preset conditions are met, the target is determined to be a suspicious target, and based on the first video, a behavior recognition model is used to perform behavior recognition on the suspicious target to determine whether there is an abnormal behavior event; The preset condition is met as follows: the target type includes an abnormal type, or the key feature does not exist in the image corresponding to the target; After the behavior recognition of the suspicious target based on the first video and the behavior recognition model is used to determine whether there is an abnormal behavior event, the method further includes: If an abnormal event occurs, the moving speed and the moving direction of the suspicious target are calculated by the position change of the suspicious target in the image of the adjacent time frame in the first video; Predict the movement trajectory of suspicious targets based on their direction of travel and traffic layout; According to the motion trajectory and the pre-deployed camera association map, determine the slave camera corresponding to the area that the suspicious target will pass through when moving along the motion trajectory; Obtaining a second video corresponding to the camera, and identifying the suspicious target using a behavior recognition model based on the second video to track and monitor the suspicious target; The method of identifying the suspicious target by using the behavior recognition model according to the second video includes: Determine, according to the distance between the slave camera and the core camera, a behavior recognition threshold of a behavior recognition model for performing behavior recognition on the second video; Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video; The behavior recognition model includes: a target video frame behavior recognition network, an optical flow frame behavior recognition network, an original video behavior recognition network and a decision network; The training process of the behavior recognition model includes: Acquire a sample video, wherein the sample video includes a plurality of samples and a behavior type corresponding to each sample image, wherein the sample includes an original video, an optical flow frame extracted based on the original video, and a target video frame where a suspicious target is determined based on the original video; Inputting the target video frame of the sample into the target video frame behavior recognition network to obtain a first recognition result; Inputting the optical flow frame extracted based on the original video into the optical flow frame behavior recognition network to obtain a second recognition result; Input the original video into the original video behavior recognition network to obtain a third recognition result; Inputting the first recognition result, the second recognition result and the third recognition result into a decision network to obtain a recognition result; Determine a loss value according to the recognition result and the behavior type, and iteratively train the behavior recognition model according to the loss value to obtain a final behavior recognition model; Each recognition network includes: a feature extraction layer, a feature fusion layer, and a classification layer. The target video frame of the sample is input into the target video frame behavior recognition network to obtain a first recognition result; the optical flow frame extracted based on the original video is input into the optical flow frame behavior recognition network to obtain a second recognition result; the original video is input into the original video behavior recognition network to obtain a third recognition result, including: Input the target video frame of the sample into the feature extraction layer, first obtain multiple first shallow features by downsampling, then obtain the first deep feature by upsampling, and obtain the first feature by the first deep feature and the first first shallow feature through the feature fusion layer; input the first feature into the classification layer to obtain the first recognition result; The optical flow frame extracted from the original video is input into the optical flow frame behavior recognition network, multiple second shallow features are first obtained by downsampling, and then the second deep features are obtained by upsampling, and the second deep features and the first second shallow features are passed through the feature fusion layer to obtain the initial second features, and the initial second features are fused with the first features to obtain the second features; the second features are input into the classification layer to obtain the second recognition result; The original video is input into the original video behavior recognition network, and multiple third shallow features are first obtained by downsampling, and then the third deep feature is obtained by upsampling, and the third deep feature and the first third shallow feature are passed through the feature fusion layer to obtain the initial third feature, and the initial third feature is fused with the second feature to obtain the third feature; the third feature is input into the classification layer to obtain the third recognition result.

2. The effective data supervision method based on digitization according to claim 1 is characterized in that: The using the target detection model to perform target detection on the first video to obtain a detection result includes: Performing object detection on the first video using the object detection model to obtain a plurality of sub-detection results in a time series; If for a certain target, there are a continuous preset number of sub-detection results in which the target types are all normal types, then the target type for the certain target in the detection result is determined to be a normal type; otherwise, it is an abnormal type.

3. The effective data supervision method based on digitization according to claim 1 is characterized in that: Also includes: If the key feature does not exist in the target image, misidentification information is recorded, and the misidentification information includes: the number of misidentifications and the image of the misidentified target; When the number of misidentifications reaches a preset number threshold, the target detection model is trained again using a set of stored images of misidentified suspicious targets.

4. The effective data supervision method based on digitization according to claim 1 is characterized in that: Determining a behavior recognition threshold of a behavior recognition model for performing behavior recognition on a second video according to a distance between the slave camera and the core camera includes: Estimate the distance from the suspicious target to the camera based on the motion trajectory and the position of the camera. Determine the proportion of the suspicious object in the second video to the entire screen; Determine the adjustment parameter according to the distance between the slave camera and the core camera and the target data, wherein the target data is the distance from the suspicious target to the slave camera or the proportion of the suspicious target in the entire screen in the second video; According to the adjustment parameter, the preset behavior recognition threshold is modified to obtain the behavior recognition threshold of the behavior recognition model for the second video; the preset behavior recognition threshold is the recognition threshold of the behavior recognition model for the first video; Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video.

5. An effective data supervision system based on digitalization, characterized in that: include: An acquisition module, used to acquire a first video captured by a core camera; The target detection module is used to perform target detection on the first video using the target detection model to obtain a detection result, where the detection result includes: a target type and an image corresponding to the target; A feature determination module is used to determine the key features corresponding to the target type according to the correspondence between the type and the key features if the target type is a normal type; and determine whether the key features exist in the image corresponding to the target; A behavior recognition module is used to determine that a target is a suspicious target when a preset condition is met, and based on the first video, use a behavior recognition model to perform behavior recognition on the suspicious target to determine whether there is an abnormal behavior event; The preset condition is met as follows: the target type includes an abnormal type, or the key feature does not exist in the image corresponding to the target; Also includes: A calculation module, configured to calculate the movement speed and the forward direction of the suspicious target by the position change of the suspicious target in the images of adjacent time frames in the first video if an abnormal event occurs; Prediction module, used to predict the movement trajectory of suspicious targets based on the forward direction and traffic layout; A slave camera determination module is used to determine the slave camera corresponding to the area that the suspicious target will pass through when moving along the motion trajectory according to the motion trajectory and the pre-deployed camera association map; A tracking and monitoring module, which is used to obtain a second video corresponding to the camera, and identify the suspicious target based on the second video using a behavior recognition model to track and monitor the suspicious target; Among them, the tracking and monitoring module is used to: Determine, according to the distance between the slave camera and the core camera, a behavior recognition threshold of a behavior recognition model for performing behavior recognition on the second video; Based on the behavior recognition threshold, the behavior recognition model is used to perform behavior recognition on the suspicious target in the second video; The behavior recognition model includes: a target video frame behavior recognition network, an optical flow frame behavior recognition network, an original video behavior recognition network and a decision network; Also included: Training modules for: Acquire a sample video, wherein the sample video includes a plurality of samples and a behavior type corresponding to each sample image, wherein the sample includes an original video, an optical flow frame extracted based on the original video, and a target video frame where a suspicious target is determined based on the original video; Inputting the target video frame of the sample into the target video frame behavior recognition network to obtain a first recognition result; Inputting the optical flow frame extracted based on the original video into the optical flow frame behavior recognition network to obtain a second recognition result; Input the original video into the original video behavior recognition network to obtain a third recognition result; Inputting the first recognition result, the second recognition result and the third recognition result into a decision network to obtain a recognition result; Determine a loss value according to the recognition result and the behavior type, and iteratively train the behavior recognition model according to the loss value to obtain a final behavior recognition model; Each recognition network includes: feature extraction layer, feature fusion layer, classification layer, The training modules are used to: Input the target video frame of the sample into the feature extraction layer, first obtain multiple first shallow features by downsampling, then obtain the first deep feature by upsampling, and obtain the first feature by the first deep feature and the first first shallow feature through the feature fusion layer; input the first feature into the classification layer to obtain the first recognition result; The optical flow frame extracted from the original video is input into the optical flow frame behavior recognition network, multiple second shallow features are first obtained by downsampling, and then the second deep features are obtained by upsampling, and the second deep features and the first second shallow features are passed through the feature fusion layer to obtain the initial second features, and the initial second features are fused with the first features to obtain the second features; the second features are input into the classification layer to obtain the second recognition result; The original video is input into the original video behavior recognition network, and multiple third shallow features are first obtained by downsampling, and then the third deep feature is obtained by upsampling, and the third deep feature and the first third shallow feature are passed through the feature fusion layer to obtain the initial third feature, and the initial third feature is fused with the second feature to obtain the third feature; the third feature is input into the classification layer to obtain the third recognition result.

6. An electronic device, characterized in that: It includes: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: execute the steps of the digital-based effective data supervision method according to any one of claims 1 to 4.

Images