Visual algorithm model development and reasoning method and device for urban governance

Abstract

The invention provides a visual algorithm model development and reasoning method and device for urban governance. The method comprises the steps of defining services and problems; collecting related data according to the business and the problem, and labeling the collected related data through a data labeling platform to obtain a data set; training a visual algorithm model based on a YOLO framework by using the data set, performing pruning, algorithm chip adaptation, model evaluation and optimization iteration on the trained visual algorithm model, and deploying the visual algorithm model into a production environment; and inputting image data to be processed, preprocessing the image data to be processed by using the visual reasoning engine, inputting the preprocessed image data into the visual algorithm model for feature extraction, performing reasoning and decision making according to the extracted features, and outputting a final reasoning prediction result. The method has the advantages that the visual algorithm model can be improved in the aspects of model generalization ability, deployment complexity, model labeling efficiency, real-time performance, prediction accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a visual algorithm model development and reasoning method and device for urban governance. Background Art

[0002] The application of smart city visual algorithm models covers many aspects such as urban security, traffic management, environmental monitoring, public services and urban governance. These applications not only improve the efficiency and accuracy of urban management, but also improve the quality of life and happiness of citizens.

[0003] With the continuous development and popularization of artificial intelligence technology, the application prospects of smart city visual algorithm models will be broader. Through AI artificial intelligence, the level of intelligent urban governance can be improved, and the video resources of various urban departments can be fully connected and integrated. With the help of big data and deep learning technology, abnormal events in urban operations can be discovered in a timely manner. For example, common urban management events such as illegal parking, road occupation, and illegal outdoor advertising can be intelligently identified and captured; these problems can be automatically discovered, identified, and reported to the city operation center through visual algorithm model reasoning, and the event type, location, and specific description can be listed one by one. In this way, the work efficiency of urban management law enforcement personnel is greatly improved, and it is convenient to deal with problems in urban management in a timely manner. However, the existing smart city visual algorithm models still need to be improved in terms of model generalization ability, deployment complexity, model annotation efficiency, real-time performance, and prediction accuracy. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a visual algorithm model development and reasoning method and device for urban governance, so as to solve the problems that the existing visual algorithm models still need to be improved in terms of model generalization ability, deployment complexity, model annotation efficiency, real-time performance, and prediction accuracy.

[0005] In a first aspect, the present invention provides a visual algorithm model development and reasoning method for urban governance, the method comprising the following steps:

[0006] Define the business and the problem;

[0007] Collect relevant data based on the defined business and problems, and annotate the collected relevant data through the data annotation platform to obtain the data set for model training;

[0008] Use the dataset to train the visual algorithm model based on the YOLO framework, perform pruning, algorithm chip adaptation, model evaluation and optimization iteration on the trained visual algorithm model, and deploy the trained visual algorithm model to the production environment;

[0009] Input the image data to be processed, use the visual reasoning engine to preprocess the image data to be processed, input the preprocessed image data into the visual algorithm model for feature extraction, and perform reasoning and decision-making based on the extracted features, so as to output the final reasoning prediction result.

[0010] In a second aspect, the present invention provides a visual algorithm model development and reasoning device for urban governance, characterized in that the device includes a definition module, a data annotation module, a model training module, and an inference prediction module;

[0011] The definition module is used to define business and problems;

[0012] The data annotation module is used to collect relevant data according to the defined business and problems, and annotate the collected relevant data through the data annotation platform to obtain a data set for model training;

[0013] The model training module is used to train the visual algorithm model based on the YOLO framework using the data set, perform pruning, algorithm chip adaptation, model evaluation and optimization iteration on the trained visual algorithm model, and deploy the trained visual algorithm model to the production environment;

[0014] The inference prediction module is used to input the image data to be processed, use the visual inference engine to preprocess the image data to be processed, input the preprocessed image data into the visual algorithm model for feature extraction, and perform inference and decision-making based on the extracted characteristics, thereby outputting the final inference prediction result.

[0015] By adopting the above technical solution of the present invention, the visual algorithm model can be improved in terms of model generalization ability, deployment complexity, model annotation efficiency, real-time performance, prediction accuracy, etc., so as to better meet the actual use needs of urban governance; specifically, at least including:

[0016] 1. The data annotation platform supports multi-person collaboration for data annotation. It not only supports the annotation of text data, but also supports image classification, target detection, semantic segmentation and other types of data annotation. It also provides a variety of built-in templates, which can facilitate users to configure according to actual needs, thereby improving annotation efficiency and reducing the requirements for professionalism;

[0017] 2. Based on the YOLO framework, the visual algorithm model for urban governance is trained and evaluated, which improves the accuracy and efficiency of the detection process and focuses on maintaining the best balance between accuracy and speed. It is suitable for real-time target detection tasks in various application fields;

[0018] 3. The latest backbone network and neck architecture are adopted to enhance the feature extraction and object detection performance. It provides target detection networks with different resolutions and instance segmentation models based on YOLACT. At the same time, models of different scales are carefully fine-tuned to improve model performance.

[0019] 4. Supports a variety of hardware platforms from CPU to GPU, which facilitates the deployment and expansion of models in practical applications and can meet the needs of a variety of computer vision tasks from simple to complex;

[0020] 5. Visual algorithm model deployment supports a variety of mainstream machine learning frameworks, such as TensorFlow, PyTorch, TensorRT, ONNX, etc., as well as Python custom APIs, which provides great flexibility for different needs;

[0021] 6. Ability to dynamically optimize model inference tasks based on real-time workload and available hardware resources (such as GPU and CPU); this resource management strategy not only improves hardware utilization, but also ensures stable operation and fast response under high load conditions.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart of the development and reasoning method of a visual algorithm model for urban governance in Embodiment 1 of the present invention;

[0025] Figure 2 It is a structural diagram of the entire system framework involved in the present invention;

[0026] Figure 3 This is a structural diagram of a visual algorithm model development and reasoning device for urban governance in Example 2 of the present invention. DETAILED DESCRIPTION

[0027] In order to better understand the technical solution of the present invention, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0028] Before introducing the technical solution of the present invention, the entire system framework involved in the present invention is first introduced and explained: Figure 2As shown in the figure, the entire system framework includes a visual reasoning application platform, a visual reasoning service, a data annotation platform, a model training platform, a visual reasoning engine, a model reasoning engine gateway, and a hardware unit, among which:

[0029] Visual reasoning services provide task operation services, warning event services, algorithm model services, computing power monitoring services, video analysis services, etc.; the task operation service supports the creation and management of computing tasks, and also supports the start, stop and view of computing task logs; the warning event service is used to index warning events in the video, supports efficient search of semi-structured or structured data such as high-dimensional vectors and attributes, and supports efficient indexing technologies such as feature quantization; the algorithm model service is used to provide algorithm models and algorithm information. The algorithm information mainly includes the video detection and analysis capabilities supported by the algorithm, the maximum number of analyzable video streams supported by the algorithm, the algorithm name and version number, etc.; the computing power monitoring service is used to realize real-time monitoring and warning of resources such as CPU, memory, GPU, etc.; the video analysis service is used to realize video analysis functions, including:

[0030] Video decoding: Use a decoder to convert video files from a compressed format to raw frame data; the decoding process can be software decoding (such as using the CPU) or hardware decoding (such as using the GPU's NVDEC unit); hardware decoding usually has higher performance and lower latency than software decoding;

[0031] Frame selection: select specific frames according to the frame extraction requirements, which can be uniform frame extraction (such as extracting N frames per second), extracting key frames (such as I frames), extracting frames at time intervals, etc.; use frame extraction algorithms or tools to filter out target frames from the decoded frame sequence;

[0032] Frame processing (optional): further processing of the selected frames, such as color space conversion (such as YUV to RGB), scaling, cropping, etc.; these processing steps may involve pixel-level calculations and are suitable for GPU acceleration;

[0033] Encoding and saving: If you need to save the frame extraction results as an image file, you need to encode the frame (such as JPEG, PNG, etc.); the encoding process can also be accelerated by GPU to improve processing efficiency; save the encoded image file to the specified storage location.

[0034] The hardware unit may include hardware resources such as CPU, GPU, and NPU.

[0035] The model inference engine gateway is used to access, encode and transcode video stream image data, and supports RTSP video stream input.

[0036] The visual reasoning engine and service are based on the YOLOv8 framework. They can use advanced video images, graphics processing technology and deep learning algorithms to build city-level artificial intelligence models. By analyzing, indexing and mining the corresponding scenes, they can empower various industry scenarios such as urban governance, security emergency response and smart communities.

[0037] Embodiment 1

[0038] This embodiment provides a visual algorithm model development and reasoning method for urban governance, such as Figure 1 As shown, the method comprises the following steps:

[0039] Step S1: Define business and issues, that is, define the business and issues involved in the urban governance process;

[0040] Step S2: Collect relevant data according to the defined business and problems, that is, collect relevant data (mainly relevant data in the urban governance process) in a targeted manner according to the defined business and problems, and annotate the collected relevant data through the data annotation platform to obtain a data set for model training;

[0041] Step S3: Use the data set to train the visual algorithm model based on the YOLO framework, perform pruning, algorithm chip adaptation, model evaluation and optimization iteration on the trained visual algorithm model, and deploy the trained visual algorithm model to the production environment;

[0042] Step S4: input the image data to be processed, use the visual reasoning engine to preprocess the image data to be processed, input the preprocessed image data into the visual algorithm model for feature extraction, and perform reasoning and decision-making based on the extracted characteristics, thereby outputting the final reasoning prediction result.

[0043] In some embodiments of the present invention, the definition of services and problems specifically includes:

[0044] Business definition: Identify business needs, set business goals based on business needs, and analyze business scenarios; business needs refer to why visual algorithm models should be developed or used, which are usually closely related to specific business scenarios or industry applications. For example, business needs can be face recognition, target detection, image segmentation, scene understanding, etc. When identifying business needs, it is also necessary to consider the potential application scenarios, user groups, and expected effects of visual algorithm models; business goals can be to improve recognition accuracy, increase processing speed, reduce false alarm rate, etc. When setting business goals, it is necessary to consider the actual situation and feasibility; business scenario analysis is to understand the actual data characteristics, environmental changes, interference factors, etc. in the business scenario. Business scenario analysis helps to better design the model framework and select the appropriate algorithm to deal with various complex situations;

[0045] Problem definition: Clarify the problem according to the business definition, decompose the clear problem into several sub-problems or steps, and understand the data characteristics; clarifying the problem means clarifying the specific problem to be solved based on the business definition. For example, in a face recognition scenario, the problem may be how to accurately identify faces in a complex background and extract key information. At the same time, when clarifying the problem, it is necessary to be specific to the input, output, and constraints of the problem; decomposing the clear problem into several sub-problems or steps will help reduce the difficulty of problem solving and improve the performance and efficiency of the model. For example, in the target detection task, the problem can be decomposed into steps such as candidate region generation, feature extraction, classification, and positioning; understanding data characteristics means in-depth understanding of the characteristics of visual data, including the source, format, size, quality, and distribution of the data, which will help to select appropriate preprocessing methods and model architectures to improve the adaptability and generalization ability of the model.

[0046] In some embodiments of the present invention, the step of labeling the collected relevant data through a data labeling platform to obtain a data set for model training specifically includes:

[0047] Preprocessing the collected relevant data includes but is not limited to: scaling the relevant image data to the size required by the visual algorithm model, enhancing the relevant image data using image enhancement technology (for example, cropping, rotating, flipping, color jittering, etc. the image data to be processed, which can increase the diversity of the data and improve the generalization ability of the model), converting the relevant image data into a format supported by the visual algorithm model, and normalizing the relevant image data;

[0048] Formulate unified annotation rules to facilitate data annotation by multiple people in specific operations and ensure the uniformity of annotation; perform annotation operations on the collected relevant data according to the annotation rules on the data annotation platform, wherein the data annotation platform provides different annotation tools according to different data types, for example, some tools are specifically used to annotate images, while others are suitable for annotating text or audio data. The present invention not only supports the annotation of text data, but also can perform multiple types of data annotation such as image classification, target detection, and semantic segmentation; the data annotation platform also provides a variety of built-in templates for users to configure, so that users can directly use built-in templates to perform configuration operations according to actual needs, thereby improving annotation efficiency and reducing the requirements for professionalism, and the data annotation platform supports data annotation by multiple people in collaboration, and the annotation operations include but are not limited to assigning classification labels, drawing bounding boxes, or transcribing text;

[0049] At the same time, the annotation results are checked during the annotation process. For example, some annotation results can be randomly selected for inspection or automatic errors can be automatically checked using automated tools to ensure the accuracy and consistency of the annotations. Alternatively, the annotation rules can be adjusted based on feedback information during the annotation process. For example, users can provide feedback information for uncertain situations during the annotation process, so that managers can adjust the annotation rules based on the feedback information or provide annotation guidance to users.

[0050] After the labeling is completed, the data is cleaned and verified to obtain a data set for model training, wherein the data set at least includes images for model training and corresponding labeling information, and the labeling information at least includes the bounding box coordinates and category labels of the target object; wherein, after the labeling is completed, the data is cleaned to remove erroneous or inconsistent labeling, and by verifying the labeled data, it can be ensured that the data set finally obtained can meet the training requirements of the model;

[0051] The obtained data set is exported into the format required for training the visual algorithm model in the YOLO framework to facilitate the use of the data set to train the visual algorithm model in the YOLO framework. At the same time, the obtained data set is divided into a training set, a test set, and a validation set. In the specific implementation of the present invention, the proportion of the training set, the test set, and the validation set can be set according to actual needs.

[0052] More specifically, the format required for training the visual algorithm model in the YOLO framework is a .txt format. For example, the annotation information is converted from XML or other formats to a .txt format dedicated to the YOLO framework, in which each line contains information such as the category, center point coordinates, width, height, etc. of the target object.

[0053] In some embodiments of the present invention, the use of a data set to train a visual algorithm model based on a YOLO framework specifically includes:

[0054] Model configuration: Download or clone the YOLO framework source code and decompress it to the local environment, set the training parameters in the configuration file (such as .yaml file) of the YOLO framework according to the model training requirements. In the specific implementation of the present invention, the training parameters may include the size of the input image, the number of categories, the number of training rounds, the learning rate and other parameters; YOLO (You Only Look Once) is an object recognition and positioning algorithm based on a deep neural network. Its biggest feature is that it runs very fast and can be used in real-time systems. Preferably, the present invention can specifically adopt the YOLOv8 framework;

[0055] Model training: Run the training script (such as train.py) provided by the YOLO framework, that is, use the training script provided by the YOLO framework to perform model training, and specify the path of the training set in the configuration file and the data set, use the training set in the data set to train the visual algorithm model based on the YOLO framework, and use the validation set in the data set to adjust the parameters of the trained visual algorithm model. When the present invention is implemented, the training script can be run in the command line, and parameters such as the configuration file and path can be specified. During the training process, the changes in training indicators (such as loss value, accuracy, etc.) can be monitored through command line output or tools such as TensorBoard, and hyperparameters such as learning rate and batch size can be adjusted in time according to the performance during the training process to optimize the training effect.

[0056] In some embodiments of the present invention, the pruning, algorithm chip adaptation, model evaluation and optimization iteration of the trained visual algorithm model specifically include:

[0057] First, prune the visual algorithm model after training or during training according to the set pruning strategy, and retrain the visual algorithm model after pruning to restore the visual algorithm model to the performance level before pruning. The pruning strategy includes unstructured pruning and structured pruning. Model pruning is a technology used to optimize model performance, which aims to reduce redundant parameters and computing load in the neural network to improve the efficiency of the model.

[0058] Unstructured pruning can include fine-grained pruning, which refers to pruning individual weights or connections. It is the smallest pruning granularity and may cause the model weight matrix to become sparse, so special hardware or software is required to support sparse operations. The method of unstructured pruning: define a threshold, reset the weights below the threshold to zero, and then recover the performance lost due to pruning by retraining the model.

[0059] Structured pruning can include vector pruning, kernel pruning and filter pruning. Vector pruning is pruning inside the convolution kernel to remove some weights but keep the structure of the convolution kernel; kernel pruning is to remove the entire convolution kernel, that is, to remove the response to the corresponding calculation channel in the input channel; filter pruning is to prune the entire convolution kernel group, which will change the number of output feature channels; the method of structured pruning is to sort based on the importance of the filter (such as L1 norm, L2 norm, etc.), and then remove the filters with lower importance. Similarly, the model needs to be retrained after pruning to restore performance;

[0060] When the present invention is implemented, the specific process of the entire pruning process includes:

[0061] Pre-training: fully train the visual algorithm model to achieve a high performance level;

[0062] Pruning: Prune the visual algorithm model according to the set pruning strategy. Pruning can be static (performed after training is completed) or dynamic (performed during training);

[0063] Fine-tuning: After pruning, the visual algorithm model needs to be retrained (fine-tuned) to restore or approach the performance before pruning. During the fine-tuning process, hyperparameters such as learning rate and training rounds can be adjusted.

[0064] Second, optimize the algorithm of the visual algorithm model according to the characteristics of the chip, and assign part of the computing tasks in the algorithm to the hardware acceleration unit supported by the chip for processing; Algorithm chip adaptation is a complex task involving model optimization, hardware characteristics consideration and adaptation process. By selecting appropriate optimization strategies, considering hardware characteristics and using appropriate adaptation tools, the model can be successfully deployed on various hardware chips and achieve efficient target detection functions;

[0065] When the present invention is implemented, the specific process of algorithm chip adaptation includes:

[0066] Algorithm requirements analysis: It is necessary to have a deep understanding of the specific requirements of the algorithm, including computational complexity, memory requirements, real-time requirements, etc. This helps determine the degree of dependence of the algorithm on hardware resources;

[0067] Chip characteristics research: Conduct in-depth research on the characteristics of the target chip, including its computing power (such as the performance of processing units such as CPU, GPU, DSP), memory size, power consumption, interface type, etc. This will help evaluate whether the chip meets the basic requirements of the algorithm;

[0068] Algorithm optimization: Optimize the algorithm according to the characteristics of the chip, which may include adjusting the algorithm structure, optimizing the data structure, parallel processing, etc., to improve the execution efficiency of the algorithm on the chip;

[0069] Hardware acceleration: If the chip supports hardware acceleration functions (such as GPU acceleration, DSP acceleration, etc.), part of the computing tasks in the algorithm can be assigned to these hardware acceleration units to further improve performance;

[0070] Interface design: Design the interface between the algorithm and the chip to ensure that the algorithm can correctly call chip resources and achieve efficient data transmission and processing.

[0071] Third, determine the evaluation indicators according to the specific application scenarios of the visual algorithm model, use the trained visual algorithm model to predict the test set in the data set to generate prediction results, and use the prediction results and the true labels of the test set to calculate the evaluation indicators, and analyze the evaluation indicators to generate evaluation results; model evaluation is specifically a process of measuring model performance through a series of methods and evaluation indicators;

[0072] When the present invention is implemented, the specific process and precautions of the entire model evaluation include:

[0073] Determine the evaluation indicators: Determine appropriate evaluation indicators based on the specific application scenarios and goals of the visual algorithm model; common evaluation indicators include accuracy, precision, recall, F1 value, ROC curve and AUC value (for binary classification problems), as well as mean square error, root mean square error, mean absolute error and R2 score (for regression problems);

[0074] Divide the data set: Divide the data set into training set, test set and validation set. The training set is used to train the model, the validation set is used to adjust the parameters (such as selecting the optimal model parameters or hyperparameters), and the test set is used to evaluate the performance of the model. It should be noted that the data set division should ensure that the data distribution between each subset is uniform as much as possible, and each subset should contain positive and negative samples to avoid overfitting or underfitting.

[0075] Calculate evaluation indicators: Use the trained visual algorithm model to predict the test set to generate prediction results, and use the prediction results and the true labels of the test set to calculate various evaluation indicators. Multiple evaluation indicators can more comprehensively reflect the performance of the model.

[0076] Beware of overfitting: If the model performs well on the training set but poorly on the test set, it may be caused by overfitting. In this case, you need to adjust the model structure, add regularization terms, or use other methods to improve the generalization ability of the model.

[0077] Cross-validation: In order to reduce the uncertainty caused by data set division, cross-validation can be used to evaluate the performance of the model. Cross-validation is to divide the data set into multiple subsets, each time using one of them as a test set and the rest as training sets, repeating this multiple times and calculating the average evaluation result.

[0078] Fourth, optimize and iterate the visual algorithm model based on the evaluation results. Model optimization and iteration is a process of continuously optimizing and improving the model based on the model evaluation and actual needs, including but not limited to:

[0079] According to the evaluation results and feedback information in specific application scenarios, identify the problems of the visual algorithm model and generate problem identification results. According to the problem identification results, iteratively optimize the visual algorithm model. The optimization methods may include adjusting the model structure, changing the learning algorithm, optimizing hyperparameters, etc. In addition, you can also try to use incremental learning or model fusion methods to improve the performance of the model.

[0080] Regularly update the sample data in the dataset, including collecting new training samples and updating the labels of old samples, to ensure that the model can adapt to data changes; use the updated dataset to retrain and evaluate the optimized visual algorithm model to verify the optimization effect;

[0081] In addition, after the optimized visual algorithm model is deployed in actual applications, feedback data can be continuously collected, and problem identification and optimization iterations can be performed again based on the feedback data to form a closed-loop feedback mechanism.

[0082] In some embodiments of the present invention, deploying the trained visual algorithm model to the production environment specifically includes:

[0083] Convert the trained visual algorithm model into a format suitable for deployment, including but not limited to ONNX, TensorRT, TensorFlow, PyTorch or CoreML format. In the specific implementation of the present invention, it is usually necessary to use a special conversion tool or library for conversion; at the same time, during the conversion process, the model can also be quantized, pruned and other optimization operations to reduce the model size and improve the inference speed;

[0084] Deploy the converted visual algorithm model to the target application so that the target application can use the visual algorithm model to make predictions and output inference prediction results;

[0085] Alternatively, the converted visual algorithm model can be deployed as a service and the corresponding service interface and protocol can be configured so that other systems or devices can call the service to achieve online reasoning and prediction.

[0086] Monitor and maintain the deployed visual algorithm model; by monitoring and maintaining the deployed visual algorithm model, you can ensure that the model runs stably in the production environment and its performance is continuously optimized. At the same time, during the monitoring process, you need to pay attention to the model's performance indicators, data set quality, hardware and software environment status, and security. In addition, you also need to formulate appropriate maintenance strategies, such as regular updates, pruning and compression, and error troubleshooting, to ensure the stability and reliability of the model. Among them:

[0087] Model monitoring specifically refers to the process of continuously monitoring and testing the performance, accuracy, stability, etc. of the model after it is deployed, mainly including:

[0088] Performance monitoring: Monitor the performance indicators of the model, such as accuracy, recall, F1 value, etc., to evaluate the prediction ability of the model; monitor the performance indicators of the model, such as response time and throughput, to ensure the efficiency of the model in practical applications;

[0089] Accuracy monitoring: Regularly check the difference between the model's predictions and actual results to assess the model's accuracy; monitor the model's performance over different time periods and data distributions to identify potential problems;

[0090] Stability monitoring: monitor the running status of the model, including memory usage, CPU occupancy, etc., to ensure that the model does not crash or anomaly during deployment; monitor the input data distribution of the model to detect potential data anomalies or data drift;

[0091] Logging: Record the model's running log and exception information for subsequent analysis and debugging;

[0092] Automated testing: Use automated testing to verify the correctness and stability of the model and avoid introducing new problems due to code modifications.

[0093] Model maintenance specifically refers to updating and optimizing the model regularly or in real time based on model monitoring to ensure the performance of the model in actual applications, mainly including:

[0094] Model update: Update the model in a timely manner according to new data and scenarios to maintain the accuracy and real-time performance of the model; when the model update fails or a problem occurs, roll back to the last available version in a timely manner to avoid affecting actual applications;

[0095] Model optimization: Perform online learning on the model to adapt to changes in data distribution; optimize the model, such as adjusting model parameters and improving model structure, to improve the generalization performance of the model;

[0096] Version control: Perform version control on the model and record the content and time of each update to facilitate model rollback and update;

[0097] Security and privacy protection: Protect the security and privacy of the model to prevent the model from being attacked or abused.

[0098] In some embodiments of the present invention, the inputting of the image data to be processed and preprocessing the image data to be processed by the visual reasoning engine specifically include:

[0099] Use libraries or APIs to obtain the image data to be processed from the data source. For example, for image files, you can use Python's Pillow library (an updated version of PIL) or OpenCV library to obtain them; for video streams or camera capture, you can use OpenCV's VideoCapture function; use the visual reasoning engine to perform preprocessing operations on the image data to be processed. The preprocessing operations include but are not limited to the following specific operations:

[0100] Scaling the image data to be processed to the size required by the visual algorithm model can usually be achieved by using an interpolation algorithm (such as bilinear interpolation, bicubic interpolation, etc.). In OpenCV, the cv2.resize() function can be used to scale the image.

[0101] Scale the pixel values ​​of the image data to be processed (such as 0-255) from the original range to the range used by the visual algorithm model during training (such as 0-1 or -1 to 1), that is, normalize the image data to be processed. Normalization helps to speed up the convergence of the model and improve the performance of the model. In Python, the NumPy library can be used to perform simple mathematical operations to achieve normalization;

[0102] Use image enhancement technology to enhance the image data to be processed. Common image enhancement technologies include random cropping, rotation, flipping, color jittering, etc. In deep learning frameworks such as OpenCV or Keras, there are ready-made image enhancement tools or APIs that can be used;

[0103] Convert the image data to be processed into a format supported by the visual algorithm model. For example, convert the image data to be processed into a NumPy array or a TensorFlow / PyTorch tensor. Of course, if the model supports batch processing of input data (i.e., processing multiple images at a time), multiple images need to be organized into batches and input into the visual algorithm model.

[0104] In a specific implementation, the present invention can improve the accuracy and robustness of feature detection by preprocessing the image data to be processed before inputting the image data into the visual algorithm model for feature extraction, thereby improving the accuracy of subsequent reasoning and prediction of the visual algorithm model.

[0105] In some embodiments of the present invention, the step of inputting the preprocessed image data into a visual algorithm model for feature extraction, and performing reasoning and decision making based on the extracted features specifically includes:

[0106] Load the trained visual algorithm model, specifically load the weights and configuration files in the visual algorithm model. These files usually include the model's parameters and architecture information;

[0107] The preprocessed image data is input into the neural network of the visual algorithm model. The neural network of the visual algorithm model uses a feature detection algorithm to perform feature detection on the preprocessed image data, and describes the detected features to generate feature descriptors, wherein the feature detection algorithm includes but is not limited to Harris corner detection algorithm, SIFT algorithm, SURF algorithm or ORB algorithm; when the present invention is implemented, it is usually necessary to convert the detected features into a vector or matrix form for representation, and this representation form can capture key information in the image, such as edges, corners, textures, etc., and enable these features to be matched and compared between different images or video frames; the feature descriptor is usually a high-dimensional vector that contains local image information around the feature point, such as gradient direction, intensity, etc. These descriptor vectors are a representation form of features, which can uniquely identify feature points in the image and be used for subsequent feature matching and task recognition;

[0108] The visual algorithm model performs forward propagation on the detected features, and uses the layer structure of the neural network in the visual algorithm model to perform layer-by-layer calculation on the detected features to generate preliminary inference prediction results; when the present invention is implemented, the forward propagation process of the visual algorithm model mainly includes data input, layer-by-layer calculation and activation, until the output layer generates the inference prediction results. First, the input data is passed to the input layer of the neural network after preprocessing. Subsequently, the data is continuously transmitted and transformed in the hidden layer through weighted summation and activation function processing. The output of each layer becomes the input of the next layer. This process is repeated until the output layer is reached. In the output layer, the inference prediction result can be obtained through the last weighted summation (possibly accompanied by an activation function). This inference prediction result can be a category label for a classification task, a continuous value for a regression task, or other types of outputs, depending on the specific design and training objectives of the neural network. Forward propagation is the basis for the neural network to make predictions and decisions, and provides the necessary input for subsequent loss calculations and backpropagation optimization.

[0109] Post-processing technology is used to post-process the preliminary inference prediction results to obtain the final inference prediction results, and decision logic is formulated based on the final inference prediction results, such as abnormal behavior detection, target tracking, etc.; among them, post-processing technology includes but is not limited to:

[0110] Non-maximum suppression (NMS): Since each neural grid may predict multiple bounding boxes, and multiple neural grids may predict the same target, the NMS algorithm is needed to remove redundant bounding boxes and retain the best one or several bounding boxes. NMS decides which bounding boxes to retain based on the confidence and overlap of the bounding boxes.

[0111] Confidence threshold filtering: Set a confidence threshold and filter out bounding boxes with confidence lower than the threshold to reduce false detections;

[0112] Category probability conversion: According to the category probability and confidence of the bounding box, the final category confidence (that is, the probability of a certain type of target existing in the bounding box) is calculated;

[0113] In addition, other post-processing operations can be performed in semantic segmentation, such as applying morphological operations to smooth boundaries and using contextual information to re-score classification results. Post-processing steps can help improve the accuracy and robustness of the model and ensure that the output results are more in line with actual needs.

[0114] At the same time, in the process of actual application of the visual algorithm model, the visual algorithm model can be further optimized and adjusted according to feedback and new data to improve its performance and generalization ability, including:

[0115] Choose the right model: Choose the right model architecture based on the task requirements (such as classification, detection, segmentation, etc.) and data characteristics, such as convolutional neural network (CNN), recurrent neural network (RNN), etc.

[0116] Structural tuning: Adjust the model's structural parameters such as the number of layers, convolution kernel size, and activation function to optimize the model's performance. You can also try using a deeper network, a larger convolution kernel, or a more efficient activation function.

[0117] Hyperparameter selection: including learning rate, batch size, regularization term, optimization algorithm, etc. These parameters have a significant impact on the training effect of the model;

[0118] Hyperparameter search: Use grid search, random search, Bayesian optimization and other methods to automatically or manually adjust hyperparameters to find the optimal parameter combination.

[0119] By adopting the above technical solution of the present invention, the visual algorithm model can be improved in terms of model generalization ability, deployment complexity, model annotation efficiency, real-time performance, prediction accuracy, etc., so as to better meet the actual use needs of urban governance; specifically, at least including:

[0120] 1. The data annotation platform supports multi-person collaboration for data annotation. It not only supports the annotation of text data, but also supports image classification, target detection, semantic segmentation and other types of data annotation. It also provides a variety of built-in templates, which can facilitate users to configure according to actual needs, thereby improving annotation efficiency and reducing the requirements for professionalism;

[0121] 2. Based on the YOLO framework, the visual algorithm model for urban governance is trained and evaluated, which improves the accuracy and efficiency of the detection process and focuses on maintaining the best balance between accuracy and speed. It is suitable for real-time target detection tasks in various application fields;

[0122] 3. The latest backbone network and neck architecture are adopted to enhance the feature extraction and object detection performance. It provides target detection networks with different resolutions and instance segmentation models based on YOLACT. At the same time, models of different scales are carefully fine-tuned to improve model performance.

[0123] 4. Supports a variety of hardware platforms from CPU to GPU, which facilitates the deployment and expansion of models in practical applications and can meet the needs of a variety of computer vision tasks from simple to complex;

[0124] 5. Visual algorithm model deployment supports a variety of mainstream machine learning frameworks, such as TensorFlow, PyTorch, TensorRT, ONNX, etc., as well as Python custom APIs, which provides great flexibility for different needs;

[0125] 6. Ability to dynamically optimize model inference tasks based on real-time workload and available hardware resources (such as GPU and CPU); this resource management strategy not only improves hardware utilization, but also ensures stable operation and fast response under high load conditions.

[0126] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0127] Embodiment 2

[0128] In this embodiment, a visual algorithm model development and reasoning device for urban governance is provided, such as Figure 3 As shown, the device includes a definition module, a data annotation module, a model training module and an inference prediction module;

[0129] The definition module is used to define businesses and issues, that is, to define businesses and issues involved in the urban governance process;

[0130] The data annotation module is used to collect relevant data according to the defined business and problems, that is, to collect relevant data (mainly relevant data in the urban governance process) in a targeted manner according to the defined business and problems, and to annotate the collected relevant data through the data annotation platform to obtain a data set for model training;

[0131] The model training module is used to train the visual algorithm model based on the YOLO framework using the data set, perform pruning, algorithm chip adaptation, model evaluation and optimization iteration on the trained visual algorithm model, and deploy the trained visual algorithm model to the production environment;

[0132] The inference prediction module is used to input the image data to be processed, use the visual inference engine to preprocess the image data to be processed, input the preprocessed image data into the visual algorithm model for feature extraction, and perform inference and decision-making based on the extracted characteristics, thereby outputting the final inference prediction result.

[0133] In some embodiments of the present invention, the definition module is specifically used to:

[0134] Business definition: Identify business needs, set business goals based on business needs, and analyze business scenarios; business needs refer to why visual algorithm models should be developed or used, which are usually closely related to specific business scenarios or industry applications. For example, business needs can be face recognition, target detection, image segmentation, scene understanding, etc. When identifying business needs, it is also necessary to consider the potential application scenarios, user groups, and expected effects of visual algorithm models; business goals can be to improve recognition accuracy, increase processing speed, reduce false alarm rate, etc. When setting business goals, it is necessary to consider the actual situation and feasibility; business scenario analysis is to understand the actual data characteristics, environmental changes, interference factors, etc. in the business scenario. Business scenario analysis helps to better design the model framework and select the appropriate algorithm to deal with various complex situations;

[0135] Problem definition: Clarify the problem according to the business definition, decompose the clear problem into several sub-problems or steps, and understand the data characteristics; clarifying the problem means clarifying the specific problem to be solved based on the business definition. For example, in a face recognition scenario, the problem may be how to accurately identify faces in a complex background and extract key information. At the same time, when clarifying the problem, it is necessary to be specific to the input, output, and constraints of the problem; decomposing the clear problem into several sub-problems or steps will help reduce the difficulty of problem solving and improve the performance and efficiency of the model. For example, in the target detection task, the problem can be decomposed into steps such as candidate region generation, feature extraction, classification, and positioning; understanding data characteristics means in-depth understanding of the characteristics of visual data, including the source, format, size, quality, and distribution of the data, which will help to select appropriate preprocessing methods and model architectures to improve the adaptability and generalization ability of the model.

[0136] In some embodiments of the present invention, in the data annotation module, the step of annotating the collected relevant data through the data annotation platform to obtain a data set for model training specifically includes:

[0137] Preprocessing the collected relevant data includes but is not limited to: scaling the relevant image data to the size required by the visual algorithm model, enhancing the relevant image data using image enhancement technology (for example, cropping, rotating, flipping, color jittering, etc. the image data to be processed, which can increase the diversity of the data and improve the generalization ability of the model), converting the relevant image data into a format supported by the visual algorithm model, and normalizing the relevant image data;

[0138] Formulate unified annotation rules to facilitate data annotation by multiple people in specific operations and ensure the uniformity of annotation; perform annotation operations on the collected relevant data according to the annotation rules on the data annotation platform, wherein the data annotation platform provides different annotation tools according to different data types, for example, some tools are specifically used to annotate images, while others are suitable for annotating text or audio data. The present invention not only supports the annotation of text data, but also can perform multiple types of data annotation such as image classification, target detection, and semantic segmentation; the data annotation platform also provides a variety of built-in templates for users to configure, so that users can directly use built-in templates to perform configuration operations according to actual needs, thereby improving annotation efficiency and reducing the requirements for professionalism, and the data annotation platform supports data annotation by multiple people in collaboration, and the annotation operations include but are not limited to assigning classification labels, drawing bounding boxes, or transcribing text;

[0139] At the same time, the annotation results are checked during the annotation process. For example, some annotation results can be randomly selected for inspection or automatic errors can be automatically checked using automated tools to ensure the accuracy and consistency of the annotations. Alternatively, the annotation rules can be adjusted based on feedback information during the annotation process. For example, users can provide feedback information for uncertain situations during the annotation process, so that managers can adjust the annotation rules based on the feedback information or provide annotation guidance to users.

[0140] After the labeling is completed, the data is cleaned and verified to obtain a data set for model training, wherein the data set at least includes images for model training and corresponding labeling information, and the labeling information at least includes the bounding box coordinates and category labels of the target object; wherein, after the labeling is completed, the data is cleaned to remove erroneous or inconsistent labeling, and by verifying the labeled data, it can be ensured that the data set finally obtained can meet the training requirements of the model;

[0141] The obtained data set is exported into the format required for training the visual algorithm model in the YOLO framework to facilitate the use of the data set to train the visual algorithm model in the YOLO framework. At the same time, the obtained data set is divided into a training set, a test set, and a validation set. In the specific implementation of the present invention, the proportion of the training set, the test set, and the validation set can be set according to actual needs.

[0142] More specifically, the format required for training the visual algorithm model in the YOLO framework is a .txt format. For example, the annotation information is converted from XML or other formats to a .txt format dedicated to the YOLO framework, in which each line contains information such as the category, center point coordinates, width, height, etc. of the target object.

[0143] In some embodiments of the present invention, in the model training module, the use of the data set to train the visual algorithm model based on the YOLO framework specifically includes:

[0144] Model configuration: Download or clone the YOLO framework source code and decompress it to the local environment, set the training parameters in the configuration file (such as .yaml file) of the YOLO framework according to the model training requirements. In the specific implementation of the present invention, the training parameters may include the size of the input image, the number of categories, the number of training rounds, the learning rate and other parameters; YOLO (You Only Look Once) is an object recognition and positioning algorithm based on a deep neural network. Its biggest feature is that it runs very fast and can be used in real-time systems. Preferably, the present invention can specifically adopt the YOLOv8 framework;

[0145] Model training: Run the training script (such as train.py) provided by the YOLO framework, that is, use the training script provided by the YOLO framework to perform model training, and specify the path of the training set in the configuration file and the data set, use the training set in the data set to train the visual algorithm model based on the YOLO framework, and use the validation set in the data set to adjust the parameters of the trained visual algorithm model. When the present invention is implemented, the training script can be run in the command line, and parameters such as the configuration file and path can be specified. During the training process, the changes in training indicators (such as loss value, accuracy, etc.) can be monitored through command line output or tools such as TensorBoard, and hyperparameters such as learning rate and batch size can be adjusted in time according to the performance during the training process to optimize the training effect.

[0146] In some embodiments of the present invention, in the model training module, the pruning, algorithm chip adaptation, model evaluation and optimization iteration of the trained visual algorithm model specifically include:

[0147] First, prune the visual algorithm model after training or during training according to the set pruning strategy, and retrain the visual algorithm model after pruning to restore the visual algorithm model to the performance level before pruning. The pruning strategy includes unstructured pruning and structured pruning. Model pruning is a technology used to optimize model performance, which aims to reduce redundant parameters and computing load in the neural network to improve the efficiency of the model.

[0148] Unstructured pruning can include fine-grained pruning, which refers to pruning individual weights or connections. It is the smallest pruning granularity and may cause the model weight matrix to become sparse, so special hardware or software is required to support sparse operations. The method of unstructured pruning: define a threshold, reset the weights below the threshold to zero, and then recover the performance lost due to pruning by retraining the model.

[0149] Structured pruning can include vector pruning, kernel pruning and filter pruning. Vector pruning is pruning inside the convolution kernel to remove some weights but keep the structure of the convolution kernel; kernel pruning is to remove the entire convolution kernel, that is, to remove the response to the corresponding calculation channel in the input channel; filter pruning is to prune the entire convolution kernel group, which will change the number of output feature channels; the method of structured pruning is to sort based on the importance of the filter (such as L1 norm, L2 norm, etc.), and then remove the filters with lower importance. Similarly, the model needs to be retrained after pruning to restore performance;

[0150] When the present invention is implemented, the specific process of the entire pruning process includes:

[0151] Pre-training: fully train the visual algorithm model to achieve a high performance level;

[0152] Pruning: Prune the visual algorithm model according to the set pruning strategy. Pruning can be static (performed after training is completed) or dynamic (performed during training);

[0153] Fine-tuning: After pruning, the visual algorithm model needs to be retrained (fine-tuned) to restore or approach the performance before pruning. During the fine-tuning process, hyperparameters such as learning rate and training rounds can be adjusted.

[0154] Second, optimize the algorithm of the visual algorithm model according to the characteristics of the chip, and assign part of the computing tasks in the algorithm to the hardware acceleration unit supported by the chip for processing; Algorithm chip adaptation is a complex task involving model optimization, hardware characteristics consideration and adaptation process. By selecting appropriate optimization strategies, considering hardware characteristics and using appropriate adaptation tools, the model can be successfully deployed on various hardware chips and achieve efficient target detection functions;

[0155] When the present invention is implemented, the specific process of algorithm chip adaptation includes:

[0156] Algorithm requirements analysis: It is necessary to have a deep understanding of the specific requirements of the algorithm, including computational complexity, memory requirements, real-time requirements, etc. This helps determine the degree of dependence of the algorithm on hardware resources;

[0157] Chip characteristics research: Conduct in-depth research on the characteristics of the target chip, including its computing power (such as the performance of processing units such as CPU, GPU, DSP), memory size, power consumption, interface type, etc. This will help evaluate whether the chip meets the basic requirements of the algorithm;

[0158] Algorithm optimization: Optimize the algorithm according to the characteristics of the chip, which may include adjusting the algorithm structure, optimizing the data structure, parallel processing, etc., to improve the execution efficiency of the algorithm on the chip;

[0159] Hardware acceleration: If the chip supports hardware acceleration functions (such as GPU acceleration, DSP acceleration, etc.), part of the computing tasks in the algorithm can be assigned to these hardware acceleration units to further improve performance;

[0160] Interface design: Design the interface between the algorithm and the chip to ensure that the algorithm can correctly call chip resources and achieve efficient data transmission and processing.

[0161] Third, determine the evaluation indicators according to the specific application scenarios of the visual algorithm model, use the trained visual algorithm model to predict the test set in the data set to generate prediction results, and use the prediction results and the true labels of the test set to calculate the evaluation indicators, and analyze the evaluation indicators to generate evaluation results; model evaluation is specifically a process of measuring model performance through a series of methods and evaluation indicators;

[0162] When the present invention is implemented, the specific process and precautions of the entire model evaluation include:

[0163] Determine the evaluation indicators: Determine appropriate evaluation indicators based on the specific application scenarios and goals of the visual algorithm model; common evaluation indicators include accuracy, precision, recall, F1 value, ROC curve and AUC value (for binary classification problems), as well as mean square error, root mean square error, mean absolute error and R2 score (for regression problems);

[0164] Divide the data set: Divide the data set into training set, test set and validation set. The training set is used to train the model, the validation set is used to adjust the parameters (such as selecting the optimal model parameters or hyperparameters), and the test set is used to evaluate the performance of the model. It should be noted that the data set division should ensure that the data distribution between each subset is uniform as much as possible, and each subset should contain positive and negative samples to avoid overfitting or underfitting.

[0165] Calculate evaluation indicators: Use the trained visual algorithm model to predict the test set to generate prediction results, and use the prediction results and the true labels of the test set to calculate various evaluation indicators. Multiple evaluation indicators can more comprehensively reflect the performance of the model.

[0166] Beware of overfitting: If the model performs well on the training set but poorly on the test set, it may be caused by overfitting. In this case, you need to adjust the model structure, add regularization terms, or use other methods to improve the generalization ability of the model.

[0167] Cross-validation: In order to reduce the uncertainty caused by data set division, cross-validation can be used to evaluate the performance of the model. Cross-validation is to divide the data set into multiple subsets, each time using one of them as a test set and the rest as training sets, repeating this multiple times and calculating the average evaluation result.

[0168] Fourth, optimize and iterate the visual algorithm model based on the evaluation results. Model optimization and iteration is a process of continuously optimizing and improving the model based on the model evaluation and actual needs, including but not limited to:

[0169] According to the evaluation results and feedback information in specific application scenarios, identify the problems of the visual algorithm model and generate problem identification results. According to the problem identification results, iteratively optimize the visual algorithm model. The optimization methods may include adjusting the model structure, changing the learning algorithm, optimizing hyperparameters, etc. In addition, you can also try to use incremental learning or model fusion methods to improve the performance of the model.

[0170] Regularly update the sample data in the dataset, including collecting new training samples and updating the labels of old samples, to ensure that the model can adapt to data changes; use the updated dataset to retrain and evaluate the optimized visual algorithm model to verify the optimization effect;

[0171] In addition, after the optimized visual algorithm model is deployed in actual applications, feedback data can be continuously collected, and problem identification and optimization iterations can be performed again based on the feedback data to form a closed-loop feedback mechanism.

[0172] In some embodiments of the present invention, in the model training module, deploying the trained visual algorithm model to the production environment specifically includes:

[0173] Convert the trained visual algorithm model into a format suitable for deployment, including but not limited to ONNX, TensorRT, TensorFlow, PyTorch or CoreML format. In the specific implementation of the present invention, it is usually necessary to use a special conversion tool or library for conversion; at the same time, during the conversion process, the model can also be quantized, pruned and other optimization operations to reduce the model size and improve the inference speed;

[0174] Deploy the converted visual algorithm model to the target application so that the target application can use the visual algorithm model to make predictions and output inference prediction results;

[0175] Alternatively, the converted visual algorithm model can be deployed as a service and the corresponding service interface and protocol can be configured so that other systems or devices can call the service to achieve online reasoning and prediction.

[0176] Monitor and maintain the deployed visual algorithm model; by monitoring and maintaining the deployed visual algorithm model, you can ensure that the model runs stably in the production environment and its performance is continuously optimized. At the same time, during the monitoring process, you need to pay attention to the model's performance indicators, data set quality, hardware and software environment status, and security. In addition, you also need to formulate appropriate maintenance strategies, such as regular updates, pruning and compression, and error troubleshooting, to ensure the stability and reliability of the model. Among them:

[0177] Model monitoring specifically refers to the process of continuously monitoring and testing the performance, accuracy, stability, etc. of the model after it is deployed, mainly including:

[0178] Performance monitoring: Monitor the performance indicators of the model, such as accuracy, recall, F1 value, etc., to evaluate the prediction ability of the model; monitor the performance indicators of the model, such as response time and throughput, to ensure the efficiency of the model in practical applications;

[0179] Accuracy monitoring: Regularly check the difference between the model's predictions and actual results to assess the model's accuracy; monitor the model's performance over different time periods and data distributions to identify potential problems;

[0180] Stability monitoring: monitor the running status of the model, including memory usage, CPU occupancy, etc., to ensure that the model does not crash or anomaly during deployment; monitor the input data distribution of the model to detect potential data anomalies or data drift;

[0181] Logging: Record the model's running log and exception information for subsequent analysis and debugging;

[0182] Automated testing: Use automated testing to verify the correctness and stability of the model and avoid introducing new problems due to code modifications.

[0183] Model maintenance specifically refers to updating and optimizing the model regularly or in real time based on model monitoring to ensure the performance of the model in actual applications, mainly including:

[0184] Model update: Update the model in a timely manner according to new data and scenarios to maintain the accuracy and real-time performance of the model; when the model update fails or a problem occurs, roll back to the last available version in a timely manner to avoid affecting actual applications;

[0185] Model optimization: Perform online learning on the model to adapt to changes in data distribution; optimize the model, such as adjusting model parameters and improving model structure, to improve the generalization performance of the model;

[0186] Version control: Perform version control on the model and record the content and time of each update to facilitate model rollback and update;

[0187] Security and privacy protection: Protect the security and privacy of the model to prevent the model from being attacked or abused.

[0188] In some embodiments of the present invention, in the inference prediction module, the input of the image data to be processed and the preprocessing of the image data to be processed by the visual inference engine specifically include:

[0189] Use libraries or APIs to obtain the image data to be processed from the data source. For example, for image files, you can use Python's Pillow library (an updated version of PIL) or OpenCV library to obtain them; for video streams or camera capture, you can use OpenCV's VideoCapture function; use the visual reasoning engine to perform preprocessing operations on the image data to be processed. The preprocessing operations include but are not limited to the following specific operations:

[0190] Scaling the image data to be processed to the size required by the visual algorithm model can usually be achieved by using an interpolation algorithm (such as bilinear interpolation, bicubic interpolation, etc.). In OpenCV, the cv2.resize() function can be used to scale the image.

[0191] Scale the pixel values ​​of the image data to be processed (such as 0-255) from the original range to the range used by the visual algorithm model during training (such as 0-1 or -1 to 1), that is, normalize the image data to be processed. Normalization helps to speed up the convergence of the model and improve the performance of the model. In Python, the NumPy library can be used to perform simple mathematical operations to achieve normalization;

[0192] Use image enhancement technology to enhance the image data to be processed. Common image enhancement technologies include random cropping, rotation, flipping, color jittering, etc. In deep learning frameworks such as OpenCV or Keras, there are ready-made image enhancement tools or APIs that can be used;

[0193] Convert the image data to be processed into a format supported by the visual algorithm model. For example, convert the image data to be processed into a NumPy array or a TensorFlow / PyTorch tensor. Of course, if the model supports batch processing of input data (i.e., processing multiple images at a time), multiple images need to be organized into batches and input into the visual algorithm model.

[0194] In a specific implementation, the present invention can improve the accuracy and robustness of feature detection by preprocessing the image data to be processed before inputting the image data into the visual algorithm model for feature extraction, thereby improving the accuracy of subsequent reasoning and prediction of the visual algorithm model.

[0195] In some embodiments of the present invention, in the inference prediction module, the step of inputting the preprocessed image data into the visual algorithm model for feature extraction, and performing inference and decision making based on the extracted features specifically includes:

[0196] Load the trained visual algorithm model, specifically load the weights and configuration files in the visual algorithm model. These files usually include the model's parameters and architecture information;

[0197] The preprocessed image data is input into the neural network of the visual algorithm model. The neural network of the visual algorithm model uses a feature detection algorithm to perform feature detection on the preprocessed image data, and describes the detected features to generate feature descriptors, wherein the feature detection algorithm includes but is not limited to Harris corner detection algorithm, SIFT algorithm, SURF algorithm or ORB algorithm; when the present invention is implemented, it is usually necessary to convert the detected features into a vector or matrix form for representation, and this representation form can capture key information in the image, such as edges, corners, textures, etc., and enable these features to be matched and compared between different images or video frames; the feature descriptor is usually a high-dimensional vector that contains local image information around the feature point, such as gradient direction, intensity, etc. These descriptor vectors are a representation form of features, which can uniquely identify feature points in the image and be used for subsequent feature matching and task recognition;

[0198] The visual algorithm model performs forward propagation on the detected features, and uses the layer structure of the neural network in the visual algorithm model to perform layer-by-layer calculation on the detected features to generate preliminary inference prediction results; when the present invention is implemented, the forward propagation process of the visual algorithm model mainly includes data input, layer-by-layer calculation and activation, until the output layer generates the inference prediction results. First, the input data is passed to the input layer of the neural network after preprocessing. Subsequently, the data is continuously transmitted and transformed in the hidden layer through weighted summation and activation function processing. The output of each layer becomes the input of the next layer. This process is repeated until the output layer is reached. In the output layer, the inference prediction result can be obtained through the last weighted summation (possibly accompanied by an activation function). This inference prediction result can be a category label for a classification task, a continuous value for a regression task, or other types of outputs, depending on the specific design and training objectives of the neural network. Forward propagation is the basis for the neural network to make predictions and decisions, and provides the necessary input for subsequent loss calculations and backpropagation optimization.

[0199] Post-processing technology is used to post-process the preliminary inference prediction results to obtain the final inference prediction results, and decision logic is formulated based on the final inference prediction results, such as abnormal behavior detection, target tracking, etc.; among them, post-processing technology includes but is not limited to:

[0200] Non-maximum suppression (NMS): Since each neural grid may predict multiple bounding boxes, and multiple neural grids may predict the same target, the NMS algorithm is needed to remove redundant bounding boxes and retain the best one or several bounding boxes. NMS decides which bounding boxes to retain based on the confidence and overlap of the bounding boxes.

[0201] Confidence threshold filtering: Set a confidence threshold and filter out bounding boxes with confidence lower than the threshold to reduce false detections;

[0202] Category probability conversion: According to the category probability and confidence of the bounding box, the final category confidence (that is, the probability of a certain type of target existing in the bounding box) is calculated;

[0203] In addition, other post-processing operations can be performed in semantic segmentation, such as applying morphological operations to smooth boundaries and using contextual information to re-score classification results. Post-processing steps can help improve the accuracy and robustness of the model and ensure that the output results are more in line with actual needs.

[0204] At the same time, in the process of actual application of the visual algorithm model, the visual algorithm model can be further optimized and adjusted according to feedback and new data to improve its performance and generalization ability, including:

[0205] Choose the right model: Choose the right model architecture based on the task requirements (such as classification, detection, segmentation, etc.) and data characteristics, such as convolutional neural network (CNN), recurrent neural network (RNN), etc.

[0206] Structural tuning: Adjust the model's structural parameters such as the number of layers, convolution kernel size, and activation function to optimize the model's performance. You can also try using a deeper network, a larger convolution kernel, or a more efficient activation function.

[0207] Hyperparameter selection: including learning rate, batch size, regularization term, optimization algorithm, etc. These parameters have a significant impact on the training effect of the model;

[0208] Hyperparameter search: Use grid search, random search, Bayesian optimization and other methods to automatically or manually adjust hyperparameters to find the optimal parameter combination.

[0209] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A visual algorithm model development and reasoning method for urban governance, characterized by: The method comprises the following steps: Define the business and the problem; Collect relevant data based on the defined business and problems, and annotate the collected relevant data through the data annotation platform to obtain the data set for model training; Use the dataset to train the visual algorithm model based on the YOLO framework, perform pruning, algorithm chip adaptation, model evaluation and optimization iteration on the trained visual algorithm model, and deploy the trained visual algorithm model to the production environment; Input the image data to be processed, use the visual reasoning engine to preprocess the image data to be processed, input the preprocessed image data into the visual algorithm model for feature extraction, and perform reasoning and decision-making based on the extracted features, so as to output the final reasoning prediction result.

2. According to claim 1, a visual algorithm model development and reasoning method for urban governance is characterized in that: The defined services and issues specifically include: Business definition: Identify business needs, set business goals based on business needs, and analyze business scenarios; Problem definition: Define the problem according to the business definition, break down the problem into several sub-problems or steps, and understand the data characteristics.

3. According to the visual algorithm model development and reasoning method for urban governance described in claim 1, it is characterized in that: The method of labeling the collected relevant data through the data labeling platform to obtain a data set for model training specifically includes: Perform pre-processing operations on the collected relevant data; Formulate unified labeling rules, and label the pre-processed relevant data according to the labeling rules on the data labeling platform, wherein the data labeling platform provides different labeling tools according to different data types, and also provides a variety of built-in templates for users to configure, and the data labeling platform supports multi-person collaboration for data labeling, and the labeling operations include but are not limited to assigning classification labels, drawing bounding boxes, or transcribing text; at the same time, check the labeling results during the labeling process or adjust the labeling rules according to the feedback information during the labeling process, and clean and verify the data after the labeling is completed, so as to obtain a data set for model training, wherein the data set at least includes images for model training and corresponding labeling information, and the labeling information at least includes the bounding box coordinates and category labels of the target object; The obtained dataset is exported into the format required for visual algorithm model training in the YOLO framework, and the obtained dataset is divided into training set, test set and validation set.

4. According to claim 3, a visual algorithm model development and reasoning method for urban governance is characterized in that: The format required for visual algorithm model training in the YOLO framework is .txt format.

5. According to claim 1, a visual algorithm model development and reasoning method for urban governance is characterized in that: The use of the data set to train the visual algorithm model based on the YOLO framework specifically includes: Model configuration: Download or clone the YOLO framework source code and decompress it to your local environment. Set the training parameters in the YOLO framework configuration file according to the model training requirements. Model training: Run the training script provided by the YOLO framework and specify the path of the configuration file and the training set in the dataset. Use the training set in the dataset to train the visual algorithm model based on the YOLO framework, and use the validation set in the dataset to adjust the parameters of the trained visual algorithm model.

6. According to claim 1, a visual algorithm model development and reasoning method for urban governance is characterized in that: The pruning of the trained visual algorithm model, algorithm chip adaptation, model evaluation and optimization iteration specifically include: Pruning the visual algorithm model after training or during training according to a set pruning strategy, and retraining the visual algorithm model after pruning to restore the visual algorithm model to the performance level before pruning, wherein the pruning strategy includes unstructured pruning and structured pruning; Optimize the algorithm of the visual algorithm model according to the characteristics of the chip, and assign part of the computing tasks in the algorithm to the hardware acceleration unit supported by the chip for processing; Determine the evaluation indicators according to the specific application scenarios of the visual algorithm model, use the trained visual algorithm model to predict the test set in the data set to generate prediction results, and use the prediction results and the true labels of the test set to calculate the evaluation indicators, and analyze the evaluation indicators to generate evaluation results; Optimize and iterate the visual algorithm model based on the evaluation results, including but not limited to: According to the evaluation results and feedback information in specific application scenarios, identify the problems existing in the visual algorithm model and generate problem identification results, and iteratively optimize the visual algorithm model based on the problem identification results; The sample data in the dataset is updated regularly, and the optimized visual algorithm model is retrained and evaluated using the updated dataset.

7. According to claim 1, a visual algorithm model development and reasoning method for urban governance is characterized in that: The deployment of the trained visual algorithm model to the production environment specifically includes: Convert the trained visual algorithm model into a format suitable for deployment, including but not limited to ONNX, TensorRT, TensorFlow, PyTorch, or CoreML formats; Deploy the converted visual algorithm model to the target application; or deploy the converted visual algorithm model as a service and configure the corresponding service interface and protocol; Monitor and maintain the deployed visual algorithm models.

8. According to claim 1, a visual algorithm model development and reasoning method for urban governance is characterized in that: The inputting of the image data to be processed and preprocessing the image data to be processed by the visual reasoning engine specifically include: Use the library or API to obtain the image data to be processed from the data source, and use the visual reasoning engine to perform preprocessing operations on the image data to be processed. The preprocessing operations include but are not limited to the following specific operations: Scale the image data to be processed to the size required by the visual algorithm model; Scale the pixel values ​​of the image data to be processed from the original range to the range used by the visual algorithm model during training; Use image enhancement technology to enhance the image data to be processed; Convert the image data to be processed into a format supported by the visual algorithm model.

9. According to claim 1, a visual algorithm model development and reasoning method for urban governance is characterized in that: The inputting of the preprocessed image data into the visual algorithm model for feature extraction, and making inferences and decisions based on the extracted features specifically include: Load the trained visual algorithm model, input the preprocessed image data into the neural network of the visual algorithm model, and the neural network of the visual algorithm model uses a feature detection algorithm to perform feature detection on the preprocessed image data, and describes the detected features to generate feature descriptors, wherein the feature detection algorithm includes but is not limited to Harris corner detection algorithm, SIFT algorithm, SURF algorithm or ORB algorithm; The visual algorithm model forward propagates the detected features, uses the layer structure of the neural network in the visual algorithm model to calculate the detected features layer by layer to generate preliminary inference prediction results, uses post-processing technology to post-process the preliminary inference prediction results to obtain the final inference prediction results, and formulates decision logic based on the final inference prediction results, where the post-processing technology includes but is not limited to non-maximum suppression, confidence threshold filtering, or category probability conversion.

10. A visual algorithm model development and reasoning device for urban governance, characterized in that: The device includes a definition module, a data annotation module, a model training module and an inference prediction module; The definition module is used to define business and problems; The data annotation module is used to collect relevant data according to the defined business and problems, and annotate the collected relevant data through the data annotation platform to obtain a data set for model training; The model training module is used to train the visual algorithm model based on the YOLO framework using the data set, perform pruning, algorithm chip adaptation, model evaluation and optimization iteration on the trained visual algorithm model, and deploy the trained visual algorithm model to the production environment; The inference prediction module is used to input the image data to be processed, use the visual inference engine to preprocess the image data to be processed, input the preprocessed image data into the visual algorithm model for feature extraction, and perform inference and decision-making based on the extracted characteristics, thereby outputting the final inference prediction result.

Images