City appearance detection model evaluation method and system, storage medium and electronic equipment

Abstract

The invention provides a city appearance detection model evaluation method and system, a storage medium and electronic equipment, and the method comprises the steps: building a basic city appearance detection model, inputting an original city appearance image data set into the basic city appearance detection model for training, and obtaining a trained city appearance detection model; presetting transformation parameters to establish a metamorphic relation model library; selecting a proper metamorphic relation, and using a diffusion model to act on the metamorphic relation to generate metamorphic relation data; applying the metamorphic relation data to the original city appearance image data set to obtain a test city appearance image data set; respectively inputting the original city appearance image data set and the test city appearance image data set into the trained city appearance detection model to respectively obtain an original image output result and a test image output result; comparing the original image output result with the test image output result, and performing division judgment on the original city appearance image; and adjusting and optimizing the trained city appearance detection model on the basis of a division judgment result.

Description

Technical Field

[0001] The present invention relates to the field of smart city management technology, and in particular to a city appearance detection model evaluation method, system, storage medium and electronic equipment. Background Art

[0002] In the field of urban appearance detection, the robustness of the model directly affects its application effect in urban management. Urban appearance detection models usually use the YOLO series of models to identify and locate specific targets in images, such as garbage, illegal construction, road damage, etc. However, due to the complexity of urban appearance detection tasks, the model often exhibits large performance fluctuations in the face of conditions such as lighting changes, target occlusion, image noise, and changes in camera angles. These robustness issues will lead to reduced detection accuracy in practical applications and increase the risk of missed detection or false detection.

[0003] In the existing technologies, the robustness evaluation of the city appearance detection model has the following shortcomings: 1. Lack of systematic robustness evaluation methods Current robustness evaluations are usually based on experience and experiments, but lack a systematic and standardized evaluation method. This lack of systematic evaluation may lead to unforeseen performance issues when the model is deployed in real applications.

[0004] 2. Insufficient processing of complex scenes In the task of urban appearance inspection, the complex urban environment often contains a variety of interference factors, such as crowds, vehicle occlusions, dynamic changes, etc. These factors put forward higher requirements on existing technologies, and the current urban appearance inspection model still has a lot of room for improvement when dealing with these complex scenarios. 3. Lack of robustness Although data enhancement strategies can improve the robustness of urban appearance detection models to a certain extent, these strategies are often targeted and cannot fully cover the various complex situations that may be encountered in practical applications. For example, certain specific lighting conditions or extreme weather conditions may still cause the detection accuracy of the model to drop significantly; adversarial attack methods are difficult to cover all types of adversarial attacks and cannot guarantee the robustness of the model in various adversarial scenarios. At the same time, adversarial training may cause the performance of the model to decline on normal samples, and there is a trade-off between performance and defense capabilities.

[0005] 4. Large amount of manpower cost and computing resource consumption Existing methods for improving urban appearance detection models, such as data augmentation strategies and adversarial training, usually require a lot of manpower for data labeling to ensure the generalization ability of the model in various scenarios. In addition, while defense methods such as adversarial training improve the robustness of the model, they are often accompanied by significant consumption of computing resources, such as increased training time and hardware requirements. These factors lead to an increase in overall costs and make it difficult to be widely used in resource-constrained environments. Summary of the invention In view of the deficiencies in the prior art, the present invention aims to provide a method for evaluating a city appearance detection model, aiming to solve the technical problems mentioned in the background technology.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions: A method for evaluating a city appearance detection model comprises the following steps: S10, establishing a basic city appearance detection model, and inputting an original city appearance image dataset with various city appearance problems into the basic city appearance detection model for training, so as to obtain a trained city appearance detection model, wherein the original city appearance image dataset is composed of a plurality of original city appearance images; S20, establishing a metamorphosis relationship model library, and applying the metamorphosis relationship data in the metamorphosis relationship model library to the original urban appearance image data set to obtain a test urban appearance image data set, wherein the metamorphosis relationship data includes a diffusion model; S30, inputting the original city appearance image dataset and the test city appearance image dataset into the trained city appearance detection model respectively, and obtaining the original image output result and the test image output result respectively; S40, comparing the original image output result and the test image output result, and performing a division and determination on the original city appearance image based on the comparison result and using an evaluation index; S50, based on the result of the division determination, adjusting and optimizing the trained city appearance detection model to improve the detection performance of the trained city appearance detection model.

[0007] According to one aspect of the above technical solution, in step S10, the city appearance problems include but are not limited to garbage dumping, illegal buildings, and damaged roads.

[0008] According to one aspect of the above technical solution, in step S20, the transformation relationship data includes image geometric transformation, image noise and image numerical transformation, the image geometric transformation includes image cropping, image rotation and flipping, image scaling and image shifting, the image noise includes Gaussian noise, multiplicative noise, salt and pepper noise, shot noise and occlusion, and the image numerical transformation includes brightness transformation, blur processing, contrast transformation, sharpening processing and saturation change.

[0009] According to one aspect of the above technical solution, the specific steps of step S20 include: Collecting the transformation relationship data that affects the city appearance image, and establishing a transformation relationship model library based on the transformation relationship data; The image geometric transformation, image noise and image numerical transformation in the metamorphosis relationship data are combined with the original urban appearance image data set, and the diffusion model in the metamorphosis relationship data is used to assign corresponding scene transformation to the original urban appearance image data set, so that the metamorphosis relationship data acts on the original urban appearance image data set to obtain a test urban appearance image data set.

[0010] According to one aspect of the above technical solution, the specific steps of step S30 include: The original city appearance image data set is input into the trained city appearance detection model to obtain the original image output result and the test image output result.

[0011] According to one aspect of the above technical solution, the specific steps of step S40 include: The original image output result is defined as I, and the test image output result is defined as I'; ; ; Wherein a represents the target category, x and y represent the horizontal coordinate and the vertical coordinate of the center point of the bounding box of a single original image in the original image output result, respectively, and h and w represent the length and width of the bounding box of the single original image, respectively; Performing target detection on the single original image to obtain an original target category set, performing target detection on a single test image in the test image output result to obtain a test target category, and performing a first classification on the original image output result using a first classification formula; ; Among them, A i is the original target category set, A i ' is the test target category set, CE i Represents a Boolean value. i =1, the single original image is classified as a target category detection error; Performing target category statistics on the single original image that is not included in the target category detection error and the corresponding single test image, and performing a second classification on the single original image that is not included in the target category detection error by using a second classification formula; ; in, represents the number of targets of target category α in the single original image, represents the number of targets of target category α in the single test image, γ represents the decision threshold, and when QE i =1, the single original image is classified as an object quantity detection error; Performing target bounding box detection on the single original image that is not included in the target number detection error and the corresponding single test image respectively, and performing a third classification on the single original image that is not included in the target number detection error by using a third classification formula; ; Among them, O i represents the original target bounding box detection result of the single original image, o represents a single target in the original target bounding box detection result, Represents the total number of targets in the original target bounding box detection result, o∈O i , loU represents the bounding box intersection operation formula, o' represents a single target in the test target bounding box detection result with the maximum loU value corresponding to o, θ=0.3, = 0.5, when LE i =1, the single original image is classified as a target position detection error; The original image output results and the corresponding test image output results that are not included in the target position detection error are classified as passing the degradation test, denoted as TP i =1.

[0012] According to one aspect of the above technical solution, after step S40, the method further includes: The ratio of the number of images classified as the target category detection error, the target number detection error, the target position detection error, and the number of images that passed the metamorphosis test to the total number of images T in the original image output result is calculated to reflect the generalization of the metamorphosis relationship: ; in represents the ratio of the number of single original images classified as target category detection errors to the total number of images T, represents the ratio of the number of single original images classified as target number detection errors to the total number of images T, represents the ratio of the number of the single original images classified as target position detection errors to the total number of images T, represents the ratio of the number of the single original images classified as having passed the degradation test to the total number of images T; Based on the generalization of the transformation relationship, the quality of the trained city appearance detection model is evaluated by the comprehensive index of model generalization:

[0013] Among them, MR score represents the quality score of the city appearance detection model after training, represents the number of targets whose target category is α in the original target bounding box detection result, n represents the total number of target categories, represents the number of targets of target category α in the bounding box detection result of the test target, Indicates the number of class α targets whose bounding box intersection-over-union ratio between the original target bounding box detection result and the test target bounding box detection result is greater than 0.5.

[0014] The present invention also provides a city appearance detection model evaluation system, comprising: Construction module: used to establish a basic city appearance detection model, and input the original city appearance image data set with various city appearance problems into the basic city appearance detection model for training, so as to obtain a trained city appearance detection model, wherein the original city appearance image data set is composed of multiple original city appearance images; Transformation module: used to establish a transformation relationship model library, and apply the transformation relationship data in the transformation relationship model library to the original urban appearance image data set to obtain a test urban appearance image data set, wherein the transformation relationship data includes a diffusion model; Output module: used to input the original city appearance image data set and the test city appearance image data set into the trained city appearance detection model respectively, and obtain the original image output result and the test image output result respectively; Classification module: used for comparing the original image output result and the test image output result, and classifying and judging the original city image based on the comparison result and using the evaluation index; Optimization module: used to adjust and optimize the trained city appearance detection model based on the result of the division and judgment, so as to improve the detection performance of the trained city appearance detection model.

[0015] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the city appearance detection model evaluation method as described above is implemented.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned city appearance detection model evaluation method when executing the computer program.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. Systematic evaluation method The present invention provides a systematic robustness evaluation method, constructs a detailed evaluation process and measurement indicators based on the metamorphosis test, fills the gap in the existing robustness evaluation method of the city appearance detection model. Compared with the existing evaluation method based on experience and experiments, the present invention provides a more standardized and comprehensive evaluation system, which can help developers better understand and improve the robustness of the model; 2. Improved ability to handle complex scenarios By introducing the metamorphosis test technology, the present invention can automatically generate and verify the adaptability of the model in complex urban environments. Compared with the existing single data enhancement or adversarial sample methods, the present invention can cover a wider range of scene changes, including factors such as lighting, occlusion, and noise, thereby significantly improving the performance of the model in complex scenes; 3. Improve model robustness and performance optimization Through a sophisticated metamorphosis test method, the present invention can identify and optimize the parts that are critical to model performance, reduce unnecessary complexity, optimize the model structure without sacrificing model detection accuracy, and improve processing efficiency, making the method suitable for application scenarios that require efficient data processing and accurate target identification; 4. Reduce human and computing resource consumption The present invention effectively reduces the consumption of manpower and computing resources through an automated testing framework and a lightweight evaluation scheme. Compared with traditional methods, the automated evaluation process of the present invention can run efficiently in a resource-constrained environment, reducing development costs while improving the efficiency and accuracy of evaluation. 5. Construction of transformation relationship based on diffusion model The diffusion model is introduced to construct the metamorphosis relationship, making the generated test data more realistic and diverse. Compared with the traditional data enhancement method, the diffusion model can capture the complex correlation information within the data set and generate more diverse test images, which effectively improves the robustness of the urban appearance detection model in diverse scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the city appearance detection model evaluation method in the first embodiment of the present invention; Figure 2 It is a structural block diagram of the city appearance detection model evaluation system in the second embodiment of the present invention; Figure 3 is a structural block diagram of an electronic device in a third embodiment of the present invention; The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0019] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0020] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0022] See also Figure 1 , which is a city appearance detection model evaluation method in the first embodiment of the present invention, comprising the following steps: S10, establishing a basic city appearance detection model, and inputting an original city appearance image dataset with various city appearance problems into the basic city appearance detection model for training, so as to obtain a trained city appearance detection model, wherein the original city appearance image dataset is composed of a plurality of original city appearance images; S20, establishing a metamorphosis relationship model library, and applying the metamorphosis relationship data in the metamorphosis relationship model library to the original urban appearance image data set to obtain a test urban appearance image data set, wherein the metamorphosis relationship data includes a diffusion model; S30, inputting the original city appearance image dataset and the test city appearance image dataset into the trained city appearance detection model respectively, and obtaining the original image output result and the test image output result respectively; S40, comparing the original image output result and the test image output result, and performing a division and determination on the original city appearance image based on the comparison result and using an evaluation index; S50, based on the result of the division determination, adjusting and optimizing the trained city appearance detection model to improve the detection performance of the trained city appearance detection model.

[0023] It can be understood that the present invention has the following beneficial effects: 1. Systematic evaluation method The present invention provides a systematic robustness evaluation method, constructs a detailed evaluation process and measurement indicators based on the metamorphosis test, fills the gap in the existing robustness evaluation method of the city appearance detection model. Compared with the existing evaluation method based on experience and experiments, the present invention provides a more standardized and comprehensive evaluation system, which can help developers better understand and improve the robustness of the model; 2. Improved ability to handle complex scenarios By introducing the metamorphosis test technology, the present invention can automatically generate and verify the adaptability of the model in complex urban environments. Compared with the existing single data enhancement or adversarial sample methods, the present invention can cover a wider range of scene changes, including factors such as lighting, occlusion, and noise, thereby significantly improving the performance of the model in complex scenes; 3. Improve model robustness and performance optimization Through a sophisticated metamorphosis test method, the present invention can identify and optimize the parts that are critical to model performance, reduce unnecessary complexity, optimize the model structure without sacrificing model detection accuracy, and improve processing efficiency, making the method suitable for application scenarios that require efficient data processing and accurate target identification; 4. Reduce human and computing resource consumption The present invention effectively reduces the consumption of manpower and computing resources through an automated testing framework and a lightweight evaluation scheme. Compared with traditional methods, the automated evaluation process of the present invention can run efficiently in a resource-constrained environment, reducing development costs while improving the efficiency and accuracy of evaluation. 5. Construction of transformation relationship based on diffusion model The diffusion model is introduced to construct the metamorphosis relationship, making the generated test data more realistic and diverse. Compared with the traditional data enhancement method, the diffusion model can capture the complex correlation information within the data set and generate more diverse test images, which effectively improves the robustness of the urban appearance detection model in diverse scenarios.

[0024] In step S10, the city appearance problems include but are not limited to garbage dumping, illegal buildings, and damaged roads.

[0025] Specifically, in step S20, the transformation relationship data includes image geometric transformation, image noise and image numerical transformation, the image geometric transformation includes image cropping, image rotation and flipping, image scaling and image shifting, the image noise includes Gaussian noise, multiplicative noise, salt and pepper noise, shot noise and occlusion, and the image numerical transformation includes brightness transformation, blur processing, contrast transformation, sharpening processing and saturation change.

[0026] It can be understood that the image geometric transformation verifies the stability and robustness of the trained city appearance detection model under different image scales and perspective changes by changing the geometric structure of the city appearance image.

[0027] Specifically, the image geometric transformation includes: Image cropping: By adjusting the size and ratio of the city image, the effect of camera position or focal length changes on image acquisition in the urban environment is simulated to test whether the trained city detection model can still accurately identify targets such as garbage dumps and illegal buildings at different aspect ratios; Image rotation and flipping: Rotation and flipping operations change the directionality of the city image to simulate the installation angle of the camera or the direction change of the target object in the actual scene, and verify whether the trained city appearance detection model can identify city appearance problems such as road signs and road damage in different directions; Image zooming: The zooming operation simulates the changes in the size of objects at different camera distances in the city scene, such as the impact of camera movement on city facilities such as trash cans and billboards, to test the robustness of the trained city detection model under these changes. Image shift: Shift adjusts the position of the target in the cityscape image to simulate the change of the target position in the actual scene, such as the change of the garbage dump location, and tests the stability of the trained cityscape detection model when the target position is offset.

[0028] Image noise adds different types of noise to the cityscape images to simulate the interference factors that may appear in the real urban environment and test the anti-interference ability of the trained cityscape detection model in a noisy environment.

[0029] Specifically, image noise includes: Gaussian noise: By simulating Gaussian noise caused by factors such as uneven lighting and camera quality, the recognition accuracy of the trained city appearance detection model under different lighting conditions (such as daytime and nighttime) is tested; Multiplicative noise: simulates the multiplicative noise that may be generated during image transmission, such as the signal instability problem in urban surveillance video data transmission, and verifies the recognition performance of the trained city appearance detection model under the condition of degraded data quality; Salt and pepper noise: By adding salt and pepper noise, we simulate the random interference that may occur in the city appearance monitoring images, such as electromagnetic interference or equipment failure, and test the noise resistance of the trained city appearance detection model in such scenarios; Shot noise: simulates shot noise caused by strong light or light source fluctuations to test the accuracy of the trained city appearance detection model in identifying problems such as road damage and garbage dumping under strong sunlight; Occlusion: By adding random occlusion areas to the cityscape image, we simulate the situation where the target object is occluded in the real scene, such as trees blocking the trash can, and test the recognition ability of the trained cityscape detection model when some information is lost.

[0030] Image numerical transformation: By adjusting parameters such as brightness, contrast, sharpness and saturation of the cityscape image, the performance of the trained cityscape detection model under different image quality conditions is tested to ensure that it can adapt to a variety of ambient light and image processing conditions.

[0031] Specifically, the image numerical transformation includes: Brightness change: By adjusting the brightness of the cityscape image, simulating the changes in lighting in different time periods or weather conditions (such as cloudy days and nights), the trained cityscape detection model is tested to see whether it can accurately identify problems such as illegal buildings and road obstacles under low-light or high-light conditions. The change in image brightness is generally achieved with the help of gamma transformation. By multiplying each pixel value on the original image, the overall brightness and darkness of the image can be adjusted. The transformation method is shown in the following formula: ; In the above formula, c and μ are constants greater than zero, and the μ value is divided by 1. If μ>1, the light intensity is enhanced, and if μ<1, the light intensity is weakened. Through different μ values, the image light intensity can be adjusted. r represents the original brightness value of each pixel in the image (that is, the grayscale value or color value of the image, usually an integer, and the value range depends on different image encoding standards, the most common one is 0-255). In the gamma transform formula, r is used as the input value, and it will adjust the brightness of the image by multiplying it with the constant c. This transformation relationship can test the performance of image recognition software under strong or weak light conditions.

[0032] Blur processing: Blur the city image to simulate the image blur caused by camera shaking or focal misalignment, and test whether the trained city appearance detection model can correctly identify abnormal situations in the urban environment in the blurred surveillance video. The blurring of the image is usually achieved by the convolution operation of the image. The calculation formula is as follows: ; Where f() represents the original image, i and j represent the rows and columns of the image respectively, k and l are the width and height of the convolution window, g() represents the output image, h(k,l) represents the convolution filter operator, and a normalized mean filter is generally used as the convolution operator. The pixel value output in the new image is the average value of the pixel value in the convolution kernel size window of the original image.

[0033] Contrast change: By adjusting the contrast of the cityscape image, the effect of different materials (such as metal and glass) reflecting light is simulated to test the ability of the trained cityscape detection model to recognize targets (such as road cracks and damaged billboards) in high-contrast or low-contrast images. The image contrast can be changed by performing linear operations on the image pixels in the RGB component space of the original image using the following formula: ; Where f() represents the pixel grayscale of the original image, the converted pixel grayscale is g(i, j), and the coefficient v Affects the contrast of the image. v When >1, the contrast is enhanced. v When <1, the contrast is reduced.

[0034] Sharpening: By sharpening the cityscape image, the edge details in the image are enhanced to test the ability of the trained cityscape detection model to identify targets with clear outlines (such as street lights and traffic signs) in the urban environment.

[0035] Saturation change: By adjusting the saturation of the cityscape image, the color changes under different weather or shooting conditions are simulated, such as the impact of cloudy or overcast weather on the image color. This tests whether the model can accurately identify abnormalities in urban facilities when the color changes are large.

[0036] Furthermore, the specific steps of step S20 include: Collecting the transformation relationship data that affects the city appearance image, and establishing a transformation relationship model library based on the transformation relationship data; The image geometric transformation, image noise and image numerical transformation in the metamorphosis relationship data are combined with the original urban appearance image data set, and the diffusion model in the metamorphosis relationship data is used to assign corresponding scene transformation to the original urban appearance image data set, so that the metamorphosis relationship data acts on the original urban appearance image data set to obtain a test urban appearance image data set.

[0037] It can be understood that the diffusion model is a type of probability-based generative model that generates data by gradually adding noise and reversing the process. Compared with the traditional generative adversarial network (GAN), the diffusion model can better capture the internal correlation information of the data set and generate more realistic and diverse image data. For the metamorphosis test in urban appearance detection, the diffusion model provides a novel method for constructing metamorphosis relations: for any input image, under the condition of keeping the content features unchanged, the recognition results of the urban appearance detection model should remain consistent after the image generated by the diffusion model is transformed. The diffusion model can be used to perform more complex transformations on the data set, such as changing the scene, perspective, lighting conditions, etc., so as to construct more diverse metamorphosis relations for comprehensive testing and verification of the urban appearance detection model.

[0038] To put it more concisely, for example, if the transformation relationship is brightness change, the diffusion model can be used to perform various complex processing based on the term "brightness change", such as brightening the light, dimming the light, etc. The present invention is the first to use the extended model as the transformation relationship data and combine it with the original urban image data set to perform urban appearance detection.

[0039] In the process of constructing the metamorphic relationship based on the diffusion model, it is necessary to first define the appropriate metamorphic relationship according to the specific scene and goal of the urban appearance detection. This "suitability" is generally based on experience. For example, in the urban environment, light intensity is an influencing factor, so light intensity can be regarded as a suitable metamorphic relationship. Subsequently, the image generation model corresponding to each metamorphic relationship is constructed by combining the relevant training data with the diffusion model. In the data generation stage, these models will generate derived test images based on the original urban appearance image dataset, simulating various changes in the urban appearance scene, such as weather changes, different shooting angles and lighting conditions, etc., to obtain a test urban appearance image dataset.

[0040] Furthermore, the specific steps of step S30 include: The original city appearance image data set is input into the trained city appearance detection model to obtain the original image output result and the test image output result.

[0041] Understandably, in order to ensure the quality of the generated images, the system will automatically compare the features of these test city image datasets with the real datasets, obtain the screened test city image datasets through quality detection means, and remove unqualified images. Finally, these screened test city image datasets will be input into the trained city appearance detection model for testing. By comparing the performance of the trained city appearance detection model on the original city appearance image dataset and the screened test city appearance image dataset, its robustness in diverse city appearance scenes is evaluated to ensure that the model can maintain high-precision detection capabilities in complex and changing environments.

[0042] Furthermore, the specific steps of step S40 include: The original image output result is defined as I, and the test image output result is defined as I'; ; ; Wherein a represents the target category, x and y represent the horizontal coordinate and the vertical coordinate of the center point of the bounding box of a single original image in the original image output result, respectively, and h and w represent the length and width of the bounding box of the single original image, respectively; Understandably, since there is no correct answer to the comparison between the original image output and the test image output, this will make the test difficult. We can only evaluate the quality of the metamorphosis test results by testing a parameter set consisting of three parameters: the type of target, the position of the target bounding box, and the confidence of the target in the original image output and the test image output. Performing target detection on the single original image to obtain an original target category set, performing target detection on a single test image in the test image output result to obtain a test target category, and performing a first classification on the original image output result using a first classification formula; ; Among them, A i is the original target category set, A i ' is the test target category set, CE i Represents a Boolean value. When CE i =1, the single original image is classified as a target category detection error; It is understandable that a single original image and a single test image often contain multiple targets. For a single original image and a single test image, if a certain target detected in the single original image is basically not detected in the transformed single test image, or a target category that is not in the single original image appears in large numbers in the transformed single test image, this article will call this single original image an object category detection error; Performing target category statistics on the single original image that is not included in the target category detection error and the corresponding single test image, and performing a second classification on the single original image that is not included in the target category detection error by using a second classification formula; ; in, represents the number of targets of target category α in the single original image, represents the number of targets of target category α in the single test image, γ represents the decision threshold, and when QE i =1, the single original image is classified as an object quantity detection error; It is understandable that if all the objects detected in a single original image are detected as objects of these categories in the transformed single test image, but the same objects in the single original image and the same objects in the transformed single test image are very different, this paper calls this single original image an object quantity detection error; Performing target bounding box detection on the single original image that is not included in the target number detection error and the corresponding single test image respectively, and performing a third classification on the single original image that is not included in the target number detection error by using a third classification formula; ; Among them, O i represents the original target bounding box detection result of the single original image, o represents a single target in the original target bounding box detection result, Represents the total number of targets in the original target bounding box detection result, o∈O i , loU represents the bounding box intersection operation formula, o' represents a single target in the test target bounding box detection result with the maximum loU value corresponding to o, θ=0.3, = 0.5, when LE i =1, the single original image is classified as a target position detection error; It is understandable that if the number of objects of the same category in a single original image and a single test image is basically the same, but the IoU of the detection boxes corresponding to each object in a single original image and a single test image is small, that is, the corresponding bounding boxes have few intersections, this article calls this image an object position detection error; The original image output results and the corresponding test image output results that are not included in the target position detection error are classified as passing the degradation test, denoted as TP i =1.

[0043] Furthermore, after step S40, the method further includes: The ratio of the number of images classified as the target category detection error, the target number detection error, the target position detection error, and the number of images that passed the metamorphosis test to the total number of images T in the original image output result is calculated to reflect the generalization of the metamorphosis relationship: ; in represents the ratio of the number of single original images classified as target category detection errors to the total number of images T, represents the ratio of the number of single original images classified as target number detection errors to the total number of images T, represents the ratio of the number of the single original images classified as target position detection errors to the total number of images T, represents the ratio of the number of the single original images classified as having passed the degradation test to the total number of images T; Based on the generalization of the transformation relationship, the quality of the trained city appearance detection model is evaluated by the comprehensive index of model generalization:

[0044] Among them, MR score represents the quality score of the city appearance detection model after training, represents the number of targets whose target category is α in the original target bounding box detection result, n represents the total number of target categories, represents the number of targets of target category α in the bounding box detection result of the test target, It represents the number of class α targets whose bounding box intersection-over-union ratio between the original target bounding box detection result and the test target bounding box detection result is greater than 0.5, and finally the average is calculated over the categories of 15 transformation parameters.

[0045] It should be noted that MR score The interval is (0.1).

[0046] Finally, based on MR score To adjust and optimize the trained city appearance detection model, such as adjusting the network structure, retraining the model, etc., to improve the detection performance under complex environmental changes.

[0047] In summary, the city appearance detection model evaluation method in the above embodiment of the present invention is specifically aimed at the city appearance detection model used in a complex urban environment. The evaluation method is implemented through a comprehensive metamorphosis relationship model library, which integrates geometric transformation, image quality adjustment and high-complexity image transformation generated by a diffusion model to simulate various visual changes in the real world. This integrated method enables the model to be tested in a wider range of real scene changes, thereby comprehensively evaluating the performance and robustness of the model; The innovation of the present invention is to use the diffusion model as part of the metamorphosis relationship model library, and use it to generate complex derivative images. These images can more realistically reflect the extreme conditions that may occur in the urban environment. Through this method, the performance of the urban appearance detection model in the face of environmental factors such as illumination changes, occlusion and image quality degradation can be systematically evaluated. The evaluation results are measured by multiple quantitative indicators such as classification error rate, target number error rate, location error rate and test pass rate, ensuring the comprehensiveness and objectivity of the evaluation. In addition, the implementation of the present invention does not rely on specific hardware and can be easily integrated into existing computer vision systems. It can also reduce manpower requirements, lower costs, and improve efficiency and accuracy through automated evaluation processes. This makes the present invention not only suitable for model evaluation in the R&D stage, but also for performance monitoring in actual operating environments. It has important practical application value for urban appearance management in smart cities.

[0048] The second embodiment of the present invention provides a city appearance detection model evaluation system, please refer to Figure 2 ,include: Construction module 11: used to establish a basic city appearance detection model, and input an original city appearance image data set with various city appearance problems into the basic city appearance detection model for training, so as to obtain a trained city appearance detection model, wherein the original city appearance image data set is composed of multiple original city appearance images; Transformation module 12: used to establish a transformation relationship model library, and apply the transformation relationship data in the transformation relationship model library to the original urban appearance image data set to obtain a test urban appearance image data set, wherein the transformation relationship data includes a diffusion model; Output module 13: used to input the original city appearance image data set and the test city appearance image data set into the trained city appearance detection model respectively, and obtain the original image output result and the test image output result respectively; Classification module 14: used to compare the original image output result and the test image output result, and to classify and determine the original city image based on the comparison result and the evaluation index; Optimization module 15: used for adjusting and optimizing the trained city appearance detection model based on the result of the division determination, so as to improve the detection performance of the trained city appearance detection model.

[0049] The third embodiment of the present invention further provides an electronic device, see Figure 3 , shown is an electronic device in the third embodiment of the present invention, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20, and when the processor 20 executes the computer program 30, the above-mentioned urban appearance detection model evaluation method is implemented.

[0050] Among them, the memory 10 includes at least one type of storage medium, and the storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10 can be an internal storage unit of an electronic device, such as a hard disk of the electronic device. In other embodiments, the memory 10 can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 10 can also include both an internal storage unit of the electronic device and an external storage device. The memory 10 can not only be used to store application software and various types of data installed in the electronic device, but also can be used to temporarily store data that has been output or is to be output.

[0051] Among them, in some embodiments, the processor 20 can be an electronic control unit (Electronic Control Unit, abbreviated as ECU, also known as a vehicle computer), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 10 or process data, such as executing access restriction programs, etc.

[0052] It should be pointed out that Figure 3 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than those shown in the figure, or combine certain components, or arrange the components differently.

[0053] The embodiment of the present invention further provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the city appearance detection model evaluation method as described above is implemented.

[0054] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0055] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0056] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0057] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0058] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for evaluating a city appearance detection model, characterized in that: The steps include: S10, establishing a basic city appearance detection model, and inputting an original city appearance image dataset with various city appearance problems into the basic city appearance detection model for training, so as to obtain a trained city appearance detection model, wherein the original city appearance image dataset is composed of a plurality of original city appearance images; S20, establishing a metamorphosis relationship model library, and applying the metamorphosis relationship data in the metamorphosis relationship model library to the original urban appearance image data set to obtain a test urban appearance image data set, wherein the metamorphosis relationship data includes a diffusion model; S30, inputting the original city appearance image dataset and the test city appearance image dataset into the trained city appearance detection model respectively, and obtaining the original image output result and the test image output result respectively; S40, comparing the original image output result and the test image output result, and performing a division and determination on the original city appearance image based on the comparison result and using an evaluation index; S50, based on the result of the division determination, adjusting and optimizing the trained city appearance detection model to improve the detection performance of the trained city appearance detection model.

2. The method for evaluating the city appearance detection model according to claim 1, characterized in that: In step S10, the city appearance problems include garbage dumping, illegal buildings, and damaged roads.

3. The method for evaluating the city appearance detection model according to claim 1, characterized in that: In step S20, the transformation relationship data includes image geometric transformation, image noise and image numerical transformation. The image geometric transformation includes image cropping, image rotation and flipping, image scaling and image shifting. The image noise includes Gaussian noise, multiplicative noise, salt and pepper noise, shot noise and occlusion. The image numerical transformation includes brightness transformation, blurring, contrast transformation, sharpening and saturation change.

4. The method for evaluating the city appearance detection model according to claim 3, characterized in that: The specific steps of step S20 include: Collecting the transformation relationship data that affects the city appearance image, and establishing a transformation relationship model library based on the transformation relationship data; The image geometric transformation, image noise and image numerical transformation in the metamorphosis relationship data are combined with the original urban appearance image data set, and the diffusion model in the metamorphosis relationship data is used to assign corresponding scene transformation to the original urban appearance image data set, so that the metamorphosis relationship data acts on the original urban appearance image data set to obtain a test urban appearance image data set.

5. The method for evaluating the city appearance detection model according to claim 4, characterized in that: The specific steps of step S30 include: Inputting the original city appearance image data set into the trained city appearance detection model to obtain an original image output result; Extracting key features of the original urban image dataset to obtain features of the real dataset; Comparing the features of the real data set with the test city appearance image data set, and removing the test city appearance images in the test city appearance image data set that do not have the features of the real data set, so as to obtain a screened test city appearance image data set; The screened test cityscape image data set is input into the trained cityscape detection model to obtain a test image output result.

6. The method for evaluating the city appearance detection model according to claim 5, characterized in that: The specific steps of step S40 include: The original image output result is defined as I, and the test image output result is defined as I'; ； ； Wherein a represents the target category, x and y represent the horizontal coordinate and the vertical coordinate of the center point of the bounding box of a single original image in the original image output result, respectively, and h and w represent the length and width of the bounding box of the single original image, respectively; Performing target detection on the single original image to obtain an original target category set, performing target detection on a single test image in the test image output result to obtain a test target category, and performing a first classification on the original image output result using a first classification formula; ； Among them, A i is the original target category set, A i ' is the test target category set, CE i Represents a Boolean value. When CE i =1, the single original image is classified as a target category detection error; Performing target category statistics on the single original image that is not included in the target category detection error and the corresponding single test image, and performing a second classification on the single original image that is not included in the target category detection error by using a second classification formula; ； in, represents the number of targets of target category α in the single original image, represents the number of targets of target category α in the single test image, γ represents the decision threshold, and when QE i =1, the single original image is classified as an object quantity detection error; Performing target bounding box detection on the single original image that is not included in the target number detection error and the corresponding single test image respectively, and performing a third classification on the single original image that is not included in the target number detection error by using a third classification formula; ； Among them, O i represents the original target bounding box detection result of the single original image, o represents a single target in the original target bounding box detection result, Represents the total number of targets in the original target bounding box detection result, o∈O i , loU represents the bounding box intersection operation formula, o' represents a single target in the test target bounding box detection result with the maximum loU value corresponding to o, θ=0.3, = 0.5, when LE i =1, the single original image is classified as a target position detection error; The original image output results and the corresponding test image output results that are not included in the target position detection error are classified as passing the degradation test, denoted as TP i =1.

7. The method for evaluating a city appearance detection model according to claim 6, characterized in that: After step S40, the method further includes: The ratio of the number of images classified as the target category detection error, the target number detection error, the target position detection error, and the number of images that passed the metamorphosis test to the total number of images T in the original image output result is calculated to reflect the generalization of the metamorphosis relationship: ； in represents the ratio of the number of single original images classified as target category detection errors to the total number of images T, represents the ratio of the number of single original images classified as target number detection errors to the total number of images T, represents the ratio of the number of the single original images classified as target position detection errors to the total number of images T, represents the ratio of the number of the single original images classified as having passed the degradation test to the total number of images T; Based on the generalization of the transformation relationship, the quality of the trained city appearance detection model is evaluated by the comprehensive index of model generalization: Among them, MR score represents the quality score of the city appearance detection model after training, represents the number of targets whose target category is α in the original target bounding box detection result, n represents the total number of target categories, represents the number of targets of target category α in the bounding box detection result of the test target, Indicates the number of class α targets whose bounding box intersection-over-union ratio between the original target bounding box detection result and the test target bounding box detection result is greater than 0.

5.

8. A city appearance detection model evaluation system, characterized in that: include: Construction module: used to establish a basic city appearance detection model, and input the original city appearance image data set with various city appearance problems into the basic city appearance detection model for training, so as to obtain a trained city appearance detection model, wherein the original city appearance image data set is composed of multiple original city appearance images; Transformation module: used to establish a transformation relationship model library, and apply the transformation relationship data in the transformation relationship model library to the original urban appearance image data set to obtain a test urban appearance image data set, wherein the transformation relationship data includes a diffusion model; Output module: used to input the original city appearance image data set and the test city appearance image data set into the trained city appearance detection model respectively, and obtain the original image output result and the test image output result respectively; Classification module: used for comparing the original image output result and the test image output result, and classifying and judging the original city image based on the comparison result and using the evaluation index; Optimization module: used to adjust and optimize the trained city appearance detection model based on the result of the division and judgment, so as to improve the detection performance of the trained city appearance detection model.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the city appearance detection model evaluation method as described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the city appearance detection model evaluation method according to any one of claims 1 to 7 is implemented.

Images