Domestic development platform Kylin V10-based group early warning and risk monitoring technology

Abstract

Aiming at dangerous behaviors and potential safety hazards of various scenes, the invention provides a group condition early warning and processing system based on a domestic development platform Kylin V10, model training is carried out through a target detection technology, pruning quantification is carried out on a model, and the model is deployed by using a reasoning framework so as to ensure that the system can efficiently run on the Kylin V10 development platform; and in cooperation with a software interface for group early warning and risk processing, group information and risk information which influence a current monitoring unit can be checked, so that monitoring personnel can carry out early warning on possible emergencies and carry out timely processing.

Description

Technical Field

[0001] This application relates to a public opinion early warning and risk monitoring technology based on the domestic development platform Kirin V10. This technology uses big data technology to collect and identify in real time public opinion information that may affect the current monitoring unit, so that monitoring personnel can give early warnings of possible emergencies. At the same time, this technology is also combined with target detection and image processing technology, which can automatically identify abnormal information in the monitoring screen and efficiently assist monitoring personnel in obtaining accurate information and handling emergencies. This technology is characterized by high efficiency, accuracy and intelligence, and has important application value in aspects such as public opinion monitoring, crisis public relations, and brand management. Background Art

[0002] With the rapid development of the Internet, the speed and scope of information dissemination are also constantly expanding. Against this background, people are increasingly concerned about the development of public opinion monitoring and processing technologies. Traditional public opinion monitoring and processing methods mainly rely on manual analysis, but this method has problems such as low efficiency and easy errors. Therefore, it is of great significance to develop a public opinion early warning and risk processing technology based on big data technology. Therefore, this application proposes a public opinion early warning and risk monitoring technology based on the domestic development platform Kirin V10. This technology uses big data technology to collect and identify in real time public opinion information that may affect the current monitoring unit, so that monitoring personnel can give early warnings of possible emergencies. At the same time, this technology is also combined with target detection and image inference technology, which can automatically identify abnormal information in the monitoring screen and efficiently assist monitoring personnel in obtaining accurate information and handling emergencies. This technology is characterized by high efficiency, accuracy and intelligence, can improve the efficiency and accuracy of monitoring, and has important application value in the security field. Summary of the Invention

[0003] This application relates to a public opinion early warning and risk monitoring technology based on the domestic development platform Kirin V10, which can analyze and give early warnings of abnormal situations in a timely manner through big data technology and image processing technology. Through this technology, monitoring personnel can discover and handle abnormal situations in the monitoring screen in a timely manner.

[0004] To achieve the above object, this application combines big data public opinion early warning technology and image processing technology, and proposes a public opinion early warning and risk monitoring technology based on the domestic development platform Kirin V10, as Figure 1 shown. First, collect abnormal event data, label and train these data, convert the trained data into a weight file in Onnx format, prune and quantize the converted weight file, and deploy the algorithm module using the Onnxruntime framework.

[0005] The method of this application mainly consists of 7 modules, as Figure 2As shown in the figure, it is divided into a model training module, a model pruning module, a model quantization module, a model deployment module, a data storage module, a video streaming and parsing module, a public opinion warning and risk monitoring module. The following will describe the seven modules in the method of this application respectively.

[0006] (1) Model Training Module

[0007] The model records the learned patterns, rules, and features after training. These patterns and rules can help the model make predictions, classifications, or perform other tasks on new input data. During the training process, the model is adjusted according to the input data to better capture the patterns and rules in the data. Therefore, it is necessary to label various dangerous behavior categories (image annotation is a process of adding labels to images, and the target range can be either using only one label for the entire image or assigning multiple labels to groups of pixels within a certain image). Adopting the rectangular box annotation method, use Labelimg to calibrate the targets in the image, generate a Txt label file with the suffix, and use the labeled dataset for training. Here, we use the Yolov5 single-stage object detection algorithm, and its model architecture is as Figure 3 .

[0008] When choosing a model, we need to consider the following issues: high precision, high speed, and ease of use. And Yolov5 perfectly meets our usage requirements and has the following advantages:

[0009] 1. High performance: Yolov5 has high accuracy and robustness in object detection tasks and can achieve faster inference speed without sacrificing too much accuracy.

[0010] 2. Flexibility: Yolov5 supports a variety of different model structures and sizes and can be selected according to specific application scenarios and hardware resources to achieve better adaptation.

[0011] 3. Ease of use: Yolov5 adopts the Pytorch framework, providing developers with a more convenient model training and deployment process, making it easier to use and customize.

[0012] 4. Efficiency: Compared with previous versions, Yolov5 has been optimized in model structure and training strategy, can more effectively utilize computing resources, and improve training and inference efficiency.

[0013] 5. Versatility: Yolov5 has good applicability in multiple fields and can be applied to multiple fields such as autonomous driving, security monitoring, and industrial inspection, with strong versatility.

[0014] The Yolov5 training process is as follows:

[0015] 1. Data Preparation: First, it is necessary to prepare the training dataset, including labeled images and corresponding annotation files. The annotation files usually include the category and location information of the targets, etc.

[0016] 2. Data Preprocessing: Preprocess the training data, including operations such as image enhancement and data augmentation, to increase the diversity and richness of the data and improve the generalization ability of the model.

[0017] 3. Model Selection: Select a suitable Yolov5 model structure and hyperparameters according to the specific application scenario and hardware resources.

[0018] 4. Loss Function Definition: Define the loss function for the object detection task, usually using a comprehensive loss function including object location loss, object category loss, etc.

[0019] 5. Model Training: Use the prepared training dataset to train the selected Yolov5 model, and continuously optimize the model parameters through the backpropagation algorithm so that the model can better fit the training data.

[0020] 6. Model Evaluation: During the training process, periodically evaluate the model using the validation set and calculate the performance metrics of the model on the validation set, such as accuracy, recall, etc.

[0021] 7. Model Tuning: According to the evaluation results of the validation set, tune the model, including adjusting the learning rate, adjusting the network structure, etc.

[0022] 8. Model Saving: When the model performs well on the validation set, save the trained model parameters for subsequent inference or deployment.

[0023] In addition, we also need to convert the trained model into a model file in the onnx format. Onnx can facilitate us to perform model optimization, including operations such as quantization, pruning, and fusion, to reduce the model size, improve the inference speed, and reduce the power consumption, so that the YOL0v5 model is more suitable for deployment in embedded devices and mobile applications.

[0024] (2) Model Pruning Module

[0025] The core idea of the model pruning technique is to minimize the storage space and computational complexity of the model while maintaining the model accuracy. Since there are often redundant and unnecessary parts in the structural units and parameters such as neurons, convolutional kernels, and weight parameters in deep learning models, the pruning technique can be used to reduce these redundant parts, thereby achieving the effect of reducing the model size and computational complexity.

[0026] Specifically, the implementation of the model pruning technique can be divided into the following steps:

[0027] 1. Initialize the model: First, a deep learning model needs to be initialized and trained to obtain a baseline model.

[0028] 2. Select pruning methods and strategies: According to specific application scenarios and requirements, select appropriate pruning methods and strategies. Common pruning methods include: structural pruning and parameter pruning, and common strategies include: global pruning and iterative pruning, etc.

[0029] 3. Prune the model: Based on the selected pruning methods and strategies, perform pruning operations on the deep learning model. Specifically, some unnecessary structural units or weight parameters can be deleted, or they can be set to 0 or a very small value.

[0030] 4. Retrain the model: Pruning operations may lead to a decrease in the accuracy of the model. Therefore, the pruned model needs to be retrained to restore the accuracy of the model.

[0031] 5. Fine-tune the model: After retraining, the model can be fine-tuned to further improve the accuracy of the model.

[0032] The model pruning algorithm we selected is Network Slimming, which has the following advantages:

[0033] 1. Model slimming: Through pruning and sparsification techniques, Network Slimming can significantly reduce the size of the neural network model, reduce the number of parameters, thereby reducing storage space occupancy and accelerating the inference speed. This is very important for deploying deep learning models on resource-constrained devices.

[0034] 2. Improved inference speed: Reducing the model size can reduce the computational amount during the inference process and accelerate the inference speed. This is very beneficial for real-time applications and model inference on edge devices.

[0035] 3. Resource conservation: The slimmed-down model requires less memory and computational resources and can run more efficiently in resource-constrained environments such as embedded systems, mobile devices, and edge devices.

[0036] 4. Performance retention: During the process of pruning and sparsification, Network Slimming usually maintains the performance of the model through methods such as fine-tuning. Therefore, the accuracy can be maintained while reducing the model size to a certain extent.

[0037] 5. Interpretability: Through pruning and sparsification, the structure of the model can be made clearer and more interpretable, eliminating some redundant connections and parameters, which helps to understand the working principle of the model.

[0038] Network Slimming has become a very attractive model compression and acceleration technology due to its advantages such as model slimming, improved inference speed, resource conservation, maintaining performance and interpretability. It is particularly suitable for deploying deep learning models in resource-constrained environments.

[0039] The specific method of Network Slimming is as follows: Figure 4 As shown, L1 regularization is imposed on the scaling factors in the BN layer, and then the scaling factors of the BN layer are continuously adjusted using L1 regularization. By making the scaling factors tend to 0, unimportant channels can be identified. Since each scaling factor corresponds to a specific convolutional channel (or neuron in the fully connected layer), this helps to discriminate and prune unimportant channels in subsequent operations. The additional regularization term has a negligible impact on the model performance, and it can even help the model to train to a higher accuracy. Pruning unimportant channels may temporarily cause a loss of performance, but this effect can be corrected through subsequent finetuning. After pruning, compared with the original network, the generated pruned network will be more compact in size, running time, and computational operations.

[0040] (3) Model Quantization Module

[0041] The main purpose of model quantization is to reduce the model size, improve the inference speed, and lower the power consumption. Especially in embedded devices and mobile applications, these factors are particularly important. Quantization can convert the parameters and activation values in the model from floating-point numbers to fixed-point numbers or integers with a lower bitwidth, thereby reducing the memory and computational resources required for model storage and calculation.

[0042] Specifically, the main advantages of model quantization are as follows:

[0043] 1. Reducing the model size: Quantization can significantly reduce the storage space of the model because the space occupied by fixed-point numbers or integers with a lower bitwidth is much smaller than that of floating-point numbers.

[0044] 2. Improving the inference speed: The quantized model can perform calculations using fixed-point numbers or integers with a lower bitwidth, which can achieve faster inference on hardware accelerators, thus improving the inference speed.

[0045] 3. Lowering the power consumption: The quantized model requires fewer computational resources and memory, so it can reduce the power consumption in embedded devices and mobile applications and extend the battery life of the device.

[0046] 4. Model quantization can significantly reduce the model size, improve the inference speed, and lower the power consumption without sacrificing too much accuracy, making the model more suitable for deployment and application in embedded devices and mobile applications.

[0047] Here we use Noise injection pseudo quantization in Qat-aware quantization. Noise injection pseudo quantization proposes a new QAT algorithm as Figure 5 , called Noise Injection Pseudo Quantization (Nipq). Nipq is based on Pqn and aims to automatically adjust quantization hyperparameters ically. When we train a network with a penalty loss term using Nipd, all network parameters (such as layer-by-layer bitwidth and quantization interval) are jointly optimized, and the instability is induced by Ste approximation. In addition, the Qat process of Nipd essentially regularizes the sum of the Hessian traces of the neural network, enabling the network to withstand additional noise with minimal quality degradation.

[0048] In summary, Nipd has four representative advantages:

[0049] 1. Both activation and weights are quantized based on a unified framework.

[0050] 2. All quantization hyperparameters (such as layer-by-layer bitwidth and quantization interval) are jointly optimized.

[0051] 3. Enhances the robustness of the network, making the optimized network easier to deploy in practice.

[0052] 4. Nipd shows state-of-the-art accuracy at the lowest cost in various applications.

[0053] (4) Model Deployment Module

[0054] Before model deployment, we need to consider requirements in multiple aspects to ensure that the deployed model can run properly in actual applications and achieve the expected results. The following are some common model deployment requirements:

[0055] 1. Hardware requirements: Model deployment needs to consider the hardware performance of the target device, including requirements for CPU, GPU, memory, storage, etc. Different models may have different requirements for hardware, so it is necessary to select a suitable hardware environment according to the characteristics of the model.

[0056] 2. Software requirements: The software environment required for deploying the model is also very important, including the operating system, library files, dependencies, etc. Ensure that the required software is installed on the target device and can run the model properly.

[0057] 3. Performance requirements: Model deployment needs to consider the performance requirements of the model, including inference speed, memory occupancy, power consumption, etc. When selecting the deployment method and optimizing the model, it is necessary to comprehensively consider the performance requirements in order to achieve good performance in actual applications.

[0058] 4. Maintainability Requirements: The deployed model needs to consider maintainability, including model updates, monitoring, troubleshooting, etc., to ensure that the model can run stably in the long term.

[0059] 5. Integration Requirements: If the model needs to be integrated into an existing application, the integration method and interface requirements with the application need to be considered to facilitate smooth integration into the application.

[0060] Taking the above requirements into comprehensive consideration, we choose to use the 0nnxruntime model deployment framework method and optimization strategy to ensure that the deployed model can achieve good results in practical applications.

[0061] Our model algorithm is divided into two parts: the inference algorithm and the tracking algorithm.

[0062] The inference algorithm is responsible for performing object detection and classification inference on the input data using the optimized model. The inference process is as follows:

[0063] 1. Model Loading: The inference algorithm is responsible for loading the optimized model, including the model structure and parameters.

[0064] 2. Data Preprocessing: Preprocess the input data, including operations such as data normalization, standardization, and scaling, to match the input requirements of the model.

[0065] 3. Inference Calculation: Use the loaded model to perform inference or prediction on the preprocessed data to obtain the output result.

[0066] 4. Post-Processing: Post-process the inference results, including operations such as decoding, inverse normalization, and parsing, to obtain the final prediction result.

[0067] 5. Output Result: Return the final prediction result to the caller or application.

[0068] Our inference algorithm can detect the following categories in total: holding a knife, holding a stick, holding a gun, injured, smoke, flame, person, conflict area, motorcycle, bicycle, car, bus, car, drone, waving an arm, kicking a leg, a total of 16 categories. These categories cover various common safety and monitoring scenarios, enabling our algorithm to be widely applied in monitoring systems.

[0069] The main function of the target tracking module is to continuously track one or more targets and obtain the desired information from the tracking process.

[0070] When choosing a tracking algorithm, we need to consider: morphological changes, scale changes, occlusion and disappearance, and image blur. To solve these problems, we need to select a tracking algorithm that can solve these problems simultaneously. The ByteTrack algorithm, a tracking method based on the Tracking-by-Detection paradigm, is a perfect choice, and it has the following advantages:

[0071] 1. High efficiency: Adopting a pixel-level tracking method, it processes images faster and has high real-time performance.

[0072] 2. Robustness: It can accurately track targets in complex environments such as target occlusion and illumination changes, and has strong robustness.

[0073] 3. Precision: It can accurately track targets at the pixel level, capture the subtle movements and changes of targets, and perform well in applications that require high-precision tracking.

[0074] Most multi-object tracking methods obtain target IDs by associating detection boxes with a correlation score higher than a threshold. For targets with a low detection score, such as occluded targets, they will be simply discarded, which brings non-negligible problems, including a large number of missed detections and fragmented trajectories. To solve this problem, the authors of the Byte Track algorithm proposed a simple, efficient and general data association method BYTE, which tracks by associating each detection box rather than just high-score detection boxes. For low-score detection boxes, their similarity with the trajectory is used to recover the real target and filter out background detections.

[0075] The main idea of the Byte Track algorithm is to create tracking trajectories, and then use the tracking trajectories to match the targets in each frame, frame by frame to match the targets, thus forming a complete trajectory.

[0076] When starting to scan the first frame, there are no any trajectories at this time: the algorithm will create trajectory objects for all target boxes and store them. Note: All the created trajectories will be marked as tracked trajectories at this time.

[0077] Starting from the second frame, the algorithm will gradually construct trajectories, and the steps are as follows, as Figure 6 :

[0078] 1. Classify the tracked trajectories and bounding boxes

[0079] ● Classify all the tracked trajectories into two categories: active and inactive (active tracks target boxes that have been tracked for more than two frames (including the trajectories newly created by the target boxes in the first frame))

[0080] ● Classify all the current frame bounding boxes into two categories: high-score and low-score (classified according to the score threshold of the bounding box (the default is 0.7))

[0081] 2. Perform the first tracking on the trajectories (only for high-score matching of active trajectories)

[0082] ● Combine all the tracked trajectories and lost-track trajectories, and call them preliminary tracking trajectories

[0083] ● Predict the possible positions and sizes of the next-frame bounding boxes of the preliminary tracking trajectories (use Kalman filter to predict the bounding boxes)

[0084] ● According to the Iou loss matrix, use the Hungarian algorithm to match the preliminary tracking trajectories and the high-score bounding boxes in the current frame, and obtain three results: the matched trajectories and bounding boxes, the trajectories that failed to be successfully matched, and the bounding boxes in the current frame that failed to be successfully matched. (The Hungarian algorithm can perform one-to-one matching between pairs according to the loss matrix and return the results of successful and unsuccessful matches)

[0085] ● Update the preliminary tracking trajectories using the successfully matched bounding boxes in the current frame (change the bounding boxes in the preliminary tracking trajectories to the bounding boxes in the current frame, and the id remains the original id)

[0086] 3. Perform the second tracking on the trajectories (only for low-score matching of active trajectories)

[0087] ● Find out the trajectories that were not matched in the first match, and filter out the tracked trajectories among them (because low-score matching does not match those trajectories that have already been lost-tracked)

[0088] ● Calculate the Iou between the above trajectories and the low-score bounding boxes in the current frame

[0089] ● Use the Hungarian algorithm to match the above tracking trajectories and the low-score bounding boxes in the current frame

[0090] ● Update the above tracking trajectories using the successfully matched bounding boxes in the current frame

[0091] ● Mark the trajectories that have not been successfully tracked yet as lost-track trajectories

[0092] 4. Track the trajectories in the unactivated state

[0093] ● Find out the bounding boxes in the current frame that were not successfully matched in the first step (the high-score bounding boxes that were not matched), and find out the unactivated trajectories

[0094] ● Calculate the Iou between the above trajectories and the bounding boxes in the current frame

[0095] ● Use the Hungarian algorithm to match the above tracking trajectories and the bounding boxes

[0096] ● Update the above tracking trajectories using the successfully matched bounding boxes in the current frame

[0097] ● At this time, the unactivated tracks that have been successfully tracked are directly marked as deleted tracks

[0098] 5. New track

[0099] ● If there is no high-score bounding box that has been successfully matched yet, it is considered a newly emerged tracking target, and a new track and a new ID will be assigned to it

[0100] 6. Return results

[0101] ● At this time, all the tracked tracks are returned. All tracks have a unique ID, and this result is used as the tracking result for each frame. We have developed four additional functions based on the results returned by the Byte Track tracking algorithm, which are:

[0102] ● Speed prediction: This functional module is used to predict the traveling speed of the target object or individual. Usually, the speed is calculated by analyzing the position changes of the target object in consecutive frame images. This function is very useful in fields such as video surveillance and traffic management, and can help the system predict the movement trajectory and future position of the target object

[0103] ● Traveling direction: The traveling direction module is used to determine the movement direction of the target object or individual. By analyzing the position changes and movement trajectory of the target in consecutive frame images, the system can infer the movement direction of the target

[0104] ● Walking trajectory: This functional module is used to track the movement trajectory of the target object or individual. Usually, the trajectory is determined by analyzing the position changes of the target in consecutive frame images

[0105] ● Aggregation detection: The aggregation detection module is used to detect the aggregation situation of the target objects or individuals in space. By analyzing the position information of the targets, the system can discover the aggregation situation of the targets in a specific area

[0106] (5) Data storage module

[0107] When inferring the video and performing target tracking, these processed data need to be saved. At this time, we need an efficient, fast, and secure database to save the data. Mysql is the best choice. It has the following advantages:

[0108] 1. Open source and free: Mysql is open source software and can be obtained and used for free. This reduces the costs of enterprises and individuals, making it a very popular choice

[0109] 2. Cross-platform support: Mysql can run on multiple operating systems, including Windows, Linux, Mac, etc. This makes it very flexible and able to adapt to different deployment environments

[0110] 3. High performance: MySQL is characterized by high performance and can process a large amount of data and complex queries quickly. It employs various optimization techniques such as indexing and query optimization to improve the performance of the database.

[0111] 4. Scalability: MySQL supports technologies such as master-slave replication, partitioned tables, and clustering, enabling horizontal and vertical scaling of the database to meet the growing data and user demands.

[0112] 5. Security: MySQL provides multiple security features such as user authentication, permission management, and data encryption to protect the security of the database, preventing unauthorized access and data leakage.

[0113] 6. Community support: MySQL has a large developer community and user base. Users can obtain rich technical documentation, tutorials, and community support to solve problems and get help.

[0114] A total of 7 data tables are created for the model algorithm and public opinion entry: Parser, Variable, Database, Opinion summary, Early stage opinion, Development stage opinion, and Containment stage opinion. The following introduces the data formats and functions of these 7 data tables:

[0115] Parser: Stores the inference and tracking algorithm call parameters

[0116] ● Id: Records the Id number of this column of information

[0117] ● Source: Gives the video or streaming address for the inference algorithm

[0118] ● Classes: Used to set the detection classes of the inference algorithm

[0119] ● Track: Used to set whether to enable object tracking

[0120] ● Id speed: Used to set the on / off of the speed prediction function in object tracking

[0121] ● Treading track: Used to set the on / off of the object walking trajectory function in object tracking

[0122] ● Direction: Used to set the on / off of the traveling direction function in object tracking

[0123] ● Social distance: Used to set the on / off of the crowd detection function in object tracking

[0124] ●Risk index: Stores the types of risks and the risk levels for each type of risk

[0125] Variable: Records the data that needs to be saved during reasoning and target tracking

[0126] ●id: Records the Id number of this column of information

[0127] ●The up: Records the number of people walking in the upward direction at the same time

[0128] ●The below: Records the number of people walking in the downward direction at the same time

[0129] ●The left: Records the number of people walking in the left direction at the same time

[0130] ●The right: Records the number of people walking in the right direction at the same time

[0131] ●Motorbike: Records the number of motorbikes at the same time

[0132] ●Bicycle: Records the number of bicycles at the same time

[0133] ●Car: Records the number of cars at the same time

[0134] ●Bus: Records the number of buses at the same time

[0135] ●Truck: Records the number of trucks at the same time

[0136] ●Person: Records the number of pedestrians at the same time

[0137] ●Escape: Records the number of pedestrians in a fleeing or chasing state at the same time

[0138] ●Armed: Records the number of people holding weapons at the same time

[0139] ●Casualties: Records the number of pedestrians in an injured or bleeding state at the same time

[0140] ●Fireworks: Records the number of smoke or fire sources at the same time

[0141] ●Crowd: Records the degree of crowding at the same time

[0142] Database: Records abnormal events discovered by the reasoning or tracking algorithm

[0143] ●Id: Records the Id number of this column of information

[0144] ●Camera id: Records under which camera this abnormal event occurred

[0145] ●Begin time: Records the start time of this abnormal event

[0146] ●Stop time: Records the end time of this abnormal event

[0147] ●Armed: Records whether anyone was armed during this abnormal event

[0148] ●Risk: Records the risk level of this abnormal event

[0149] ●Description: Records the description of this abnormal event

[0150] ●Locartion: Records the actual geographical location of this abnormal event

[0151] Opinion summary: Used to summarize public opinion information at each stage, including the total table Id, and the Ids of public opinion at the corresponding budding stage, development stage, and containment stage

[0152] ●Id: Records the Id number of this column of information

[0153] ●Early stage id: Budding stage public opinion Id, associated with the budding stage public opinion table

[0154] ●Development stage id: Development stage public opinion Id, associated with the development stage public opinion table

[0155] ●Containment stage id: Containment stage public opinion Id, associated with the containment stage public opinion table

[0156] Early stage opinion: Records public opinion information in the budding stage

[0157] ●Id: Records the Id number of this column of information

[0158] ●Start time: Records the occurrence time of the public opinion event

[0159] ●Person: Records the number of people participating in this event

[0160] ●Place: Records the location information where the public opinion event occurred

[0161] ●Category: Records the category of the public opinion event

[0162] ●Describe: Used to describe the content of the public opinion event in detail

[0163] ●Spread number: Records the spread degree of public sentiment events

[0164] ●Pathway: Records the dissemination pathway of public sentiment events

[0165] ●Source: Records the source of obtaining public sentiment information

[0166] Development stage opinion: Records public sentiment information in the development stage

[0167] ●Id: Records the Id number of this column of information

[0168] ●More time: Records the development time of public sentiment events

[0169] ●Place: Records the location information of the occurrence of public sentiment events

[0170] ●Describe: Used to describe the content of public sentiment events in detail

[0171] ●Spread number: Records the spread degree of public sentiment events

[0172] ●Pathway: Records the dissemination pathway of public sentiment events

[0173] ●Source: Records the source of obtaining public sentiment information

[0174] Containment stage opinion: Used to record public sentiment information in the containment stage

[0175] ●Id: Records the Id number of this column of information

[0176] ●End time: Records the end time of public sentiment events

[0177] ●Infuence place: Records the regional information affected by public sentiment events

[0178] ●Describe: Used to describe the content of public sentiment events in detail;

[0179] The three data tables of Parser, Variable, and database are set from the calling program to the recording of exception information, enabling users to better control the calling of the program and also enabling users to warn of possible unexpected events.

[0180] The four tables, Opinion summary, Early Stage opinion, Development stage opinion, and Containment stage opinion, follow the three stages of the public sentiment incident, namely the germination, development, and containment, and record the core information of these three stages respectively.

[0181] (6) Video streaming and analysis module

[0182] We analyze the video during reasoning and target tracking. When an abnormal event is detected, the algorithm automatically saves the event from the beginning of the event to the end of the abnormal event. Recording the complete video of the abnormal event has the following benefits:

[0183] 1. Track the development of events: Complete video records can help track the development of events, understand the full picture of events, help analyze the causes and processes of events, and provide a basis for subsequent processing and decision-making.

[0184] 2. Ensure safety: Complete video records can help monitor and ensure safety. For some important places or sensitive areas, complete video records can provide security and help detect abnormal situations and deal with them in a timely manner.

[0185] 3. Post-event analysis: Complete video records can be used as the basis for post-event analysis. For some important events or work processes, complete video records can help with post-event analysis and summary, identify problems and improve work.

[0186] In addition, during the reasoning and target tracking process, the processed video frames are continuously pushed and streamed, which can improve the versatility of the program. As long as the surveillance video is in the same network, it can be viewed. However, before pushing the stream, we need a tool that can ensure smooth streaming. Here we choose to use the Ffmpeg open source cross-platform multimedia processing tool, which has the following features: cross-platform, multi-functional, efficient, free and open source, and customizable. Ffmpeg can meet our needs in many aspects.

[0187] Currently, our algorithm supports multiple protocols, namely: Rtsp, Rtmp, Hls, Tcp, and Udp, a total of five communication protocols, which can cover a variety of usage scenarios.

[0188] (7) Public sentiment warning and risk monitoring module

[0189] The main function of the public sentiment warning and risk monitoring module is to present the analyzed and extracted data to users in the form of an interface so as to monitor and warn of risks in a timely manner.

[0190] In the public opinion warning sub-module, we divide the data into Internet data and local data. The purpose of doing this is to monitor public opinion information more comprehensively. Internet data can help us understand the trends and events of public opinion worldwide, while local data can more accurately reflect local public opinion hotspots and events, thus improving the accuracy and timeliness of warnings. By integrating Internet data and local data, we can better grasp the public opinion situation, timely discover potential risks and take corresponding measures. Here we use the Python language to obtain the desired information from popular websites and sort and organize it. Users can select the information they want to follow. If they choose to follow a certain piece of information, the program will continuously follow up on the development of this event.

[0191] The risk monitoring sub-module displays the abnormal data detected by the algorithm to the user in the form of an interface in real time. During the real-time monitoring process, the algorithm continuously transmits the abnormal data to the database, and the risk monitoring sub-module continuously reads the data from the database to ensure the real-time nature of the data so that users can handle emergencies in a timely manner. Description of the Drawings

[0192] Figure 1 It is the overall framework diagram of the public opinion warning and risk monitoring technology based on the domestic development platform Kylin V10.

[0193] Figure 2 It is the component module diagram of the method of this application.

[0194] Figure 3 It is the structure diagram of the Yolov5 model.

[0195] Figure 4 It is the pruning flow chart of the Network Slimming algorithm.

[0196] Figure 5 It is the general overview diagram of the Noise injection pseudo quantization algorithm process

[0197] Figure 6 It is the flow chart of the Byte Track tracking algorithm.

[0198] Figure 7 It is the interface diagram of the public opinion warning and risk monitoring software

[0199] Figure 8 It is the interface diagram for adding new data of "Internet data" in the public opinion warning and risk monitoring software interface

[0200] Figure 9 It is the interface diagram of the case library in the public opinion warning and risk monitoring software interface Specific implementation method

[0202] To better describe the public sentiment early warning and risk monitoring technology based on the domestic development platform Kylin V10, the following provides the specific implementation manners of this application.

[0203] This project develops the public sentiment early warning and risk monitoring software based on the Kylinos v10 system, and designs a Chinese user interface (GUI) that is convenient for users to operate. Users can enter the public sentiment early warning and risk monitoring software by entering "ip" plus ":8080" in the browser. After entering the software, three function modules can be seen, namely: "Monitoring Video", "Public Sentiment Early Warning", and "Case Library", as Figure 7 .

[0204] In the "Monitoring Video" module, users can select a camera or video for monitoring according to different usage requirements. The software supports multiple different video formats: four different video formats of Mp4, H254, H254, and Flv.

[0205] Before selecting the monitoring video, the settings of the software can also be debugged. There are the following three modules in the settings interface:

[0206] ● Object Detection: There are a total of 16 detectable categories in the object detection module, namely: knife, holding a stick, holding a gun, injured, smoke, flame, person, conflict area, motorcycle, bicycle, car, bus, car, drone, waving an arm, kicking a leg. Users can detect one or more detection targets.

[0207] ● Object Tracking: There are a total of four enabled functions in the object tracking module, namely: speed prediction, moving direction, walking trajectory, and predicted detection. Users can select to enable one or more functions.

[0208] ● Risk Assessment: There are a total of 19 evaluation rules in the risk assessment, namely: setting the number of people holding knives, setting the number of people holding sticks, setting the number of people holding guns, setting flames, setting smoke, setting drones, abnormal behavior (tracking and running wildly), abnormal micro-behavior (kicking a leg), abnormal micro-behavior (waving an arm), setting the number of people escaping in all directions, setting the number of people in conflict, setting the number of people lying injured, setting the input of people walking in the same direction, setting the walking speed of people, setting the number of people gathering, setting the number of pedestrians, setting the number of vehicles, setting the number of non-motor vehicles, and setting the distribution degree. Users can set each evaluation index according to their own usage situations.

[0209] In the "Public Sentiment Early Warning" module, users have two ways to obtain public sentiment information, namely "Internet data" and "local data". Users can choose the way to obtain information according to their needs.

[0210] In "Internet Data", you can view hot information according to the current time, or search for the information you want to view by: event name, keyword, information source, nature of the event, event location, and event scope. In "Internet Data", you can also construct information by yourself, such as Figure 8 , and enrich the content of the event by inputting the following information: event name, event description, keyword, event category, information source, nature of the event, event occurrence time, event end time, number of people involved in the event, event location, spread coefficient, dissemination route, and number of affected people.

[0211] In "Local Data", the data display method and the way of constructing information are generally the same as those in "Internet", but the difference is that the data viewed in "Local Data" are all manually added local data.

[0212] In the "Case Library", all risk event information is recorded, such as Figure 9 , and you can quickly query information by inputting the keyword of the event, the event occurrence time, or the risk coefficient. Each piece of information can be modified, deleted, added, etc.

[0213] In order to realize the identification of abnormal events on the public opinion early warning and risk monitoring software, this project sets up three modules: Yolov5 training module, NetworkSlimming pruning module, and quantization module.

[0214] (1) Yolov5 training module

[0215] In the dataset folder of the Yolov5 project file, there are two folders, namely images and labels. The images folder contains the train and val folders, which are the training image folder and the validation image folder respectively. The labels folder contains the train and val folders, which are the training label folder and the validation label folder respectively.

[0216] 1.yaml is the configuration file used for the model to train on the dataset, and its function is to provide appropriate parameter settings for the model training for the dataset.

[0217] train is the storage path of the training images, val is the storage path of the validation images, test is the specified image path of the test images, which can also be not specified, nc is the number of types of objects to be detected in the dataset, and names specifies the category names of the objects to be detected in the dataset.

[0218] 2. Prepare the pre-trained model and configure the training model files

[0219] The role of the pre-trained model is to provide a detection model pre-trained on a large-scale dataset, which can be used for the rapid development of object detection application programs. By training on a large-scale dataset, the pre-trained model can effectively learn general feature representations, which can be used to identify the categories of various different objects.

[0220] In the train.py file of the Yolov5 project, the def parse_opt() function records relevant command-line parameters, and performs necessary checks and processing on some command-line parameters (such as path checks, value range checks, etc.) to ensure the normal operation of the program. Next, we introduce the necessary parameters:

[0221] ● weights is the parameter name for the model weights. Generally, it refers to the pre-trained weight file that needs to be loaded into the model

[0222] ● cfg is the parameter name for the configuration file of the model structure (abbreviation of config). This parameter allows users to load a specified configuration file.

[0223] To initialize the structure of the current model

[0224] ● data is the parameter name for the configuration file of the dataset. This parameter allows users to load a specified dataset configuration file to initialize the dataset and perform model training or inference.

[0225] ● epochs refers to the number of training rounds of the model. This parameter allows users to set the number of training rounds required for the model to achieve better model performance.

[0226] ● batch size refers to the number of images input into the model in each batch. This parameter allows users to set the number of images input in each batch during model training to achieve better training effects.

[0227] (2) Network Slimming pruning module

[0228] Before pruning, we need to prepare the following files separately: the model file trained using Yolov5, the yaml file of the trained model, and put these files into the Network Slimming project file.

[0229] The Network Slimming pruning module uses the train.py file for training and pruning. The def parse_opt() function in the train.py file records relevant command-line parameters. Next, we introduce the necessary parameters:

[0230] ● "weights" is the parameter name for model weights. Generally, it refers to the pre-trained weight file that needs to be loaded into the model.

[0231] ● "data" is the parameter name for the configuration file of the dataset. This parameter allows users to load the specified dataset configuration file to initialize the dataset and perform model training or inference.

[0232] ● "epochs" refers to the number of training rounds of the model. This parameter allows users to set the number of training rounds the model needs to achieve better model performance.

[0233] ● "batch size" refers to the number of images input into the model in each batch. This parameter allows users to set the number of images input in each batch during model training to achieve better training effects.

[0234] ● "Sr" refers to the size of the model that needs to be pruned. For example, (0.7) indicates that the model will be reduced by approximately seventy percent.

[0235] ● "Oa" refers to what the pruning strategy is. The default value is: Network Slimming

[0236] (3) Quantization module

[0237] Before quantization, we need to prepare the following files: the dataset for validating model accuracy and the pruned model, and put the prepared files into the Noise injection pseudo quantization project file.

[0238] The Noise injection pseudo quantization module uses the train.py file for quantization. The relevant command-line parameters are stored in the perser variable in the train.py file. Next, we introduce the necessary parameters:

[0239] ● Data: Dataset directory

[0240] ● Model: Model name. Here we input Yolov5s

[0241] ● Lr: Learning rate: The default value is 0.4

[0242] ● Batch: Batch size

[0243] ● Epoch: Number of training rounds.

Claims

1. The public opinion early warning and risk monitoring technology based on the domestic development platform Kylin V10 is characterized in that: The specific steps are as follows: Step 1) Use the inference model Yolov5 for model training; Step 2) Use the pruning model; Step 3) Use the quantization model; Step 4) Use the model deployment framework Onnx for model deployment; Step 5) Use the Mysql database for data storage; Step 6) Use Ffmpeg and the mediamtx streaming server; Step 7) Use vue.js3 to write the interface functions.

2. The sentiment warning and risk monitoring technology based on the domestic development platform Kylin V10 according to claim 1, characterized in that Step 1) For the prepared images, use the LabelImg annotation tool for annotation. Use the inference model Yolov5 to train the annotated dataset, and convert the trained model into the Onnx format.

3. The sentiment early warning and risk monitoring technology based on the domestic development platform Kylin V10 according to claim 1, wherein Step 2).

4. The sentiment warning and risk monitoring technology based on the domestic development platform Kylin V10 according to claim 1, characterized in that Step 3).

5. The sentiment warning and risk monitoring technology based on the domestic development platform Kylin V10 according to claim 1, characterized in that Step 4) Fine-tune the model parameters of the Onnx deployment framework, and use the quantized Onnx model for inference.

6. The sentiment warning and risk monitoring technology based on the domestic development platform Kylin V10 according to claim 1, wherein Step 5) Save the data required during the inference process in the Mysql database.

7. The sentiment warning and risk monitoring technology based on the domestic development platform Kylin V10 according to claim 1, characterized in that Step 6) Use the Ffmpeg video processing tool and the mediamtx streaming server to stream the inferred video.

8. The sentiment early warning and risk monitoring technology based on the domestic development platform Kylin V10 according to claim 1, characterized in that Step 7) Use the vue.js3 framework to complete the interaction between the functions on the software interface and the algorithm.

Images