Road landscape quality evaluation method based on machine vision aesthetic attention detection

Abstract

The invention discloses a road landscape quality evaluation method based on machine vision aesthetic attention detection. The method comprises the following specific steps: S1, collecting image data of a road landscape; s2, carrying out landscape aesthetics evaluation participated by the public; s3, generating an image dichotomy label according to the score; s4, establishing a complete data set of the road landscape quality; s5, establishing a'feature data set 'of the road landscape quality; s6, using the data set to carry out two-step deep transfer learning; s7, detecting aesthetic attention by utilizing'class activation mapping '; and S8, obtaining a road landscape quality evaluation result and application. According to the invention, automatic detection of aesthetic attention is carried out in a manner of combining machine vision and public participation, and the detection is used as a basis for road landscape quality evaluation. The method is convenient to operate, the evaluation result is more visual and accurate, and the method has better public representativeness.

Description

Technical Field

[0001] The present invention relates to the field of visual landscape quality evaluation, and particularly to a method for evaluating road landscape quality based on machine vision aesthetic attention detection. Background Art

[0002] The construction of road landscapes is an important starting point for the integrated development of transportation and tourism. Its quality evaluation is an essential part of linear landscape projects such as beautiful highways and tourist scenic roads, and can provide references for various tasks such as road route selection, renovation, maintenance, and operation. On roads, people's aesthetic attention reflects potential visual attractors or pollutants in the environment. Information such as the type, distribution, and density of these objects reflects the aesthetic characteristics of this section of the road, thus constituting an important part of road landscape quality evaluation, and providing references for subsequent rest area and viewing point layout, tourism product planning, and landscape improvement and renovation projects.

[0003] Currently, capturing the visual focus of the human eye through eye tracking is a common method for investigating attention. However, this method has defects in the investigation of road landscape aesthetic attention. First, in the driving and riding scenarios on roads, people's attention includes not only aesthetic appreciation but also risk identification, sign reading, direction judgment, etc. And it is difficult for eye tracking to separate the attention results from different purposes. Second, road landscapes have a large spatial span and diverse types, and their evaluation process requires public participation to reduce individual aesthetic biases. Therefore, sufficient sample data is needed to ensure the effectiveness of the investigation and evaluation. However, the eye tracking program is complex and the conditions are harsh, and it is difficult to further expand its sample data under limited costs.

[0004] At the same time, how to identify visual attractors and pollutants in road landscapes is also an important entry point for landscape quality evaluation and landscape improvement and renovation. However, previous evaluations of road landscapes were mostly based on the overall evaluation at the "field" scale, and the quality calculation of the current scene was achieved through the evaluation of indicators such as scenic beauty, openness, green view rate, and sky visibility rate. There is a lack of investigation through the attention mechanism to lock in a certain scenery and conduct landscape quality evaluation at the "object" scale, so as to give more targeted and guiding evaluation results. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a method for evaluating road landscape quality based on machine vision aesthetic attention detection, so as to solve the problems of identification, evaluation, and statistics of landscape aesthetic attention.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] A method for evaluating road landscape quality based on machine vision aesthetic attention detection of the present invention specifically comprises the following steps:

[0008] S1. Collect image data of road landscapes: Use a car-mounted camera with an anti-shake function to take pictures from a fixed perspective in front of or on both sides of the road. The shooting interval is based on the principle of distinguishing changes in road landscapes within adjacent units and can be specifically set according to road conditions.

[0009] S2. Conduct a public participation-based aesthetic assessment of road landscapes: Compile a Likert 5-point scale and invite the public to score the collected images. The scores from 1 to 5 represent "not beautiful" to "beautiful" in sequence, and 3 points represent "uncertain". To reduce the subjective influence of individuals on the evaluation results, the number of evaluations for each photo is not less than 5 person-times.

[0010] S3. Generate binary classification labels for images based on the scores: Calculate the mean score of each photo sample and divide all data into binary classification labels of positive and negative aesthetic qualities. That is, when the photo mean is greater than 3 points, the sample is labeled as "beautiful"; otherwise, it is "not beautiful".

[0011] S4. Establish a "complete dataset" for road landscape quality: Statistically analyze the proportion of the number of positive and negative samples in all photos. If the number of positive and negative labels is unbalanced, use image enhancement methods to supplement the data with a small sample size to form a complete dataset with balanced positive and negative samples.

[0012] S5. Establish a "feature dataset" for road landscape quality: Select the same proportion of high-score and low-score samples from the complete dataset to form a "feature dataset" carrying more landscape aesthetic information.

[0013] S6. Perform two-step deep transfer learning using the dataset: Build a deep learning framework, randomly generate a training set and a validation set in a 7:3 ratio from the complete dataset, and use a pre-trained deep learning model for image processing as the source model for transfer learning. Modify the fully connected layer of the source model for a binary classification task and conduct the first deep transfer learning to obtain an initial model based on the complete dataset.

[0014] S61. On the basis of the initial model, use the "feature dataset" to conduct the second deep transfer learning to improve the effectiveness and accuracy of the final model.

[0015] S7. Detect aesthetic attention using "Class Activation Mapping (CAM)": The CAM technology identifies the importance of image regions by projecting the weights of the output layer back to the convolutional feature map. The generated heatmap shows the attention of the deep learning model. Import the road landscape photos to be detected into the final prediction model loaded with CAM in sequence to obtain the "aesthetic attention" heatmap of the image.

[0016] S8. Road Landscape Quality Assessment Results and Applications: According to the detection results, if the model determines that the imported image is a positive sample, the aesthetic attention of the model reflects the visual attractors that appear on this section of the road; if the model determines it as a negative sample, the aesthetic attention reflects the visual pollutants existing on this section of the road; evaluate the landscape quality of this section of the road based on the types and distributions of the attractors and pollutants, and propose corresponding protection, utilization, or improvement measures.

[0017] As a preferred technical solution of the present invention, in S6, a deep learning framework is built through PyTorch, and a pre-trained model related to image processing is used to carry out deep learning for the binary classification task.

[0018] As a preferred technical solution of the present invention, in S7, the CAM technology outputs the different weights shown by all pixels in the sample image when performing the aesthetic binary classification task, and attaches pixels with different weights to different depths of colors to represent the depth of the model's aesthetic attention.

[0019] As a preferred technical solution of the present invention, the camera in S1 further includes a GPS module for recording positioning information in the photo, which is convenient for the spatial connection between the evaluation results and the road points.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] The present invention uses machine vision technology based on deep learning to detect aesthetic attention, and has made great improvements in the operability of the evaluation process and the reliability of the evaluation results, specifically including the following technical effects:

[0022] First, the method proposed by the present invention conducts aesthetic evaluation in the form of public participation. By setting up photo questionnaires on the web page and mobile phone, it is convenient to invite more participants to carry out the evaluation, strengthening the public representativeness of the detection results;

[0023] Second, the image acquisition of road landscapes is relatively convenient. Any institution or individual can collect local data for transfer learning for different road environments, further strengthening the applicable scope of the model while increasing the sample size;

[0024] Third, during the evaluation process of the scale, since the sample labels are only generated based on aesthetic judgments, the attention content of the prediction model only involves the aesthetics of road landscapes, and the detection results will be faithful to aesthetic attention, excluding the interference of other attentions such as safety and wayfinding in the road environment;

[0025] Fourth, the model will give positive and negative aesthetic attentions according to the binary classification results of aesthetic labels, thereby locking in the visual attractors and pollutants in the road landscape, which is helpful for formulating protection, utilization, and improvement measures for road landscapes. Description of the Drawings

[0026] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the accompanying drawings:

[0027] Figure 1 is the technical flow chart of the present invention;

[0028] Figure 2 is a schematic diagram of the loss value and accuracy rate of the two-step deep transfer learning model of the present invention;

[0029] Figure 3 is a schematic diagram of the aesthetic attention detection result of the present invention. Specific Embodiments

[0030] The following describes the preferred embodiments of the present invention with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0031] Embodiment 1

[0032] As Figures 1-3 shown, the present invention provides a method for evaluating the quality of road landscapes based on machine vision aesthetic attention detection, which specifically includes the following steps:

[0033] S1. Use a vehicle-mounted motion camera to collect photos from the perspective directly in front of the road;

[0034] According to the degree of change of the surrounding landscapes during road driving, take a photo with GPS information every 0.5 - 1 km.

[0035] In this example, a total of about 5000 km of driving was carried out in the southwestern region of China, and 7065 effective pictures were collected, recording different landscape types such as mountains, plains, hills, forests, grasslands, lakes, riverbanks, villages, and suburbs along the way.

[0036] S2. Use the photo data to carry out a public participation aesthetic evaluation to generate landscape aesthetic labels for all samples;

[0037] In this example, through the development of a WeChat mini-program, a batch cyclic non-replacement photo scale was generated. When a user accesses the mini-program, the mini-program will randomly select 50 photos from 7065 photos to form a Likert scale of 1 - 5 points, and invite the user to rate each photo. When the number of times all photos are drawn is greater than or equal to 5 times, the scale rating session ends.

[0038] S3. The mean score of each photo will be used as the final score of the sample. When the final score is greater than 3, the sample will be labeled as "beautiful" (positive sample); otherwise, it will be labeled as "not beautiful" (negative sample). In this example, a total of 1,023 people were finally invited to rate the samples, resulting in 4,506 positive samples and 2,559 negative samples.

[0039] The specific scoring results are shown in the following table:

[0040]

[0041] Table 1. Photo Scoring Results of Public Participation

[0042] S4. Since the number of positive samples is 1,947 more than that of negative samples, a corresponding number of photos were randomly selected from the negative samples, and after horizontal flipping, they were added to the negative samples. Finally, a complete dataset with a total of 9,012 samples was formed. The training set and validation set were randomly generated from the complete dataset in a 7:3 ratio for future use.

[0043] S5. Since there are many samples with unclear aesthetic features (score 2.5 - 3.5), and the number of high-score (score ≥ 3.5) or low-score (score ≤ 2.5) samples with significant aesthetic features is small, which may lead to insufficient reliability of the model. As a solution, in this example, all samples in the top and bottom 10% of the scores were selected to form a feature dataset. The training set and validation set were randomly generated from the feature dataset in a 7:3 ratio for future use.

[0044] S6. Use PyTorch to build a deep learning framework and use a pre-trained model related to image processing to carry out deep learning for binary classification tasks. In this example, the pre-trained ResNet-18 model was used, and the fully connected layer of the model was modified to output positive and negative binary classification data. First, training was carried out based on the complete dataset. During the 30 rounds of training, the model showed a minimum average loss of 0.55 and an accuracy of 0.73 on the validation set in the 8th round. This indicates that the initial model can already roughly predict the public's aesthetic judgment of road landscapes.

[0045] S61. Based on the initial model, carry out the second deep transfer learning to enhance the sensitivity of the model to important aesthetic information. During the second 30 rounds of training, the model showed a minimum average loss of 0.32 and an accuracy of 0.88 on the validation set in the 7th round ( Figure 2 ). At this time, the final model has good reliability for the task of predicting road landscape aesthetics.

[0046] S7. Using the "Class Activation Mapping (CAM)" technology, the weight values of each pixel in the photo based on the prediction results of the final model, that is, the main attention areas of the model in the photo, are projected onto the original photo in the form of a heat map. The high-heat areas reflect the landscape aesthetic attention based on public participation evaluation. By inputting the photos of any section in the dataset or the local images of newly captured road landscapes into the model, the detection of aesthetic attention can be achieved.

[0047] S8. Import the photos of the section to be evaluated, and use the positive and negative sample classification of the model for the photos and the detection results of aesthetic attention as the basis for road landscape quality evaluation. In this example, the images collected in S1 are input. The results show that objects such as roadside tree groves, mountains, grasslands, and farmlands are often detected by the model as visual attractors that cause positive attention, while objects such as potholes on the road surface, mud, wasteland on the roadside, and buildings are detected as visual pollutants that cause negative attention. Since the photos carry GPS information, the positions and distributions of attractors or pollutants can be determined according to the detection results. Furthermore, landscape protection and utilization measures can be proposed based on the evaluation results of attractors, or landscape improvement and transformation plans can be proposed for the evaluation results of pollutants ( Figure 3 ).

[0048] The present invention relies on deep learning to develop a method for detecting landscape aesthetic attention based on machine vision for the quality evaluation of road landscapes. Compared with the eye movement tracking technology, on the one hand, this method eliminates the cumbersome process of eye movement experiments and greatly improves the convenience of evaluation operations; on the other hand, it can shield the attention of other intentions such as safety and wayfinding, and effectively obtain the attention results derived from landscape aesthetics. The technical method proposed by the present invention can increase the reliability of landscape quality evaluation and provide an evidence-based basis for the planning and construction of road landscapes.

[0049] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for evaluating the quality of road landscapes based on machine vision aesthetic attention detection, characterized in that, The specific steps are as follows: S1. Collect image data of road landscapes: Use a car-mounted camera with an anti-shake function to take pictures from a fixed perspective in front of or on both sides of the road. The shooting interval is based on the principle of distinguishing changes in road landscapes within adjacent units, and can be specifically set according to road conditions; S2. Conduct a public participation-based aesthetic assessment of road landscapes: Compile a Likert 5-point scale and invite the public to score the collected images. Scores from 1 to 5 represent "not beautiful" to "beautiful" in sequence, and 3 points represent "uncertain". To reduce the subjective influence of individuals on the assessment results, the number of evaluations for each photo is not less than 5 person-times; S3. Generate binary classification labels for images based on the scores: Calculate the mean score of each photo sample, and divide all data into binary classification labels of positive and negative aesthetic qualities, that is, when the photo mean is greater than 3 points, the sample is labeled as "beautiful", otherwise it is "not beautiful"; S4. Establish a "complete dataset" for road landscape quality: Count the proportion of the number of positive and negative samples in all photos. If the number of positive and negative labels is unbalanced, use image enhancement methods to supplement the data with a small sample size to form a complete dataset with balanced positive and negative samples; S5. Establish a "feature dataset" for road landscape quality: Select the same proportion of high-score and low-score samples from the complete dataset to form a feature dataset carrying more landscape aesthetic information; S6. Apply two-step deep transfer learning using the dataset: Build a deep learning framework, randomly generate a training set and a validation set in a 7:3 ratio from the complete dataset, and use a pre-trained deep learning model for image processing as the source model for transfer learning. Modify the fully connected layer of the source model for a binary classification task, and conduct the first deep transfer learning to obtain an initial model based on the complete dataset; S61. On the basis of the initial model, use the "feature dataset" to conduct the second deep transfer learning to improve the effectiveness and accuracy of the final model; S7. Detect aesthetic attention using "class activation mapping": The class activation mapping technology identifies the importance of image regions by projecting the weights of the output layer back to the convolutional feature map. The generated heatmap shows the attention of the deep learning model. Import the road landscape photos to be detected into the final prediction model loaded with class activation mapping in sequence to obtain the "aesthetic attention" heatmap of the image; S8. Road landscape quality assessment results and applications: According to the detection results, if the model determines that the imported image is a positive sample, the aesthetic attention of the model reflects the visual attractors that appear in this section of the road; If the model determines it to be a negative sample, the aesthetic attention reflects the visual pollutants existing in this section of the road; Evaluate the landscape quality of this section of the road according to the types and distributions of attractors and pollutants, and propose corresponding protection, utilization, or improvement measures.

2. The method for evaluating the quality of road landscape based on machine vision aesthetic attention detection according to claim 1, wherein, In S6, a deep learning framework is built through PyTorch, and a pre-trained model related to image processing is used to conduct deep learning for the binary classification task.

3. The road landscape quality assessment method based on machine vision aesthetic attention detection according to claim 1, wherein In S7, the class activation mapping technology outputs the different weights shown by all pixels in the sample image when performing the aesthetic binary classification task. Attaching pixels with different weights to different depths of colors represents the depth of the model's aesthetic attention.

4. A method for evaluating the quality of road landscapes based on machine vision aesthetic attention detection according to claim 1, characterized in that, The camera in S1 also includes a GPS module for recording positioning information in the photo and evaluating the spatial connection with the road points.

Images