A rural geographic tag updating method based on social media picture data

Abstract

The present application aims at the limitations of the prior art, and proposes a rural geographic label updating method based on social media picture data, which combines multiple deep learning methods to improve the prediction accuracy while ensuring the prediction efficiency. The method can automatically update picture labels and classify various scenes, which is widely applicable and effectively improves the rural geographic label updating efficiency, reduces the labor cost and work intensity.

Description

Technical Field

[0001] This invention relates to the field of automated processing technology for rural images, and more specifically, to a method for updating rural geotags based on social media image data. Background Technology

[0002] With the rapid advancement of internet technology and the widespread adoption of smartphones in rural areas, a large number of social media apps have become deeply integrated into villagers' daily lives. The "Village Scenery Photo Shooting" mini-program offers various functions such as image and text social interaction, questionnaire design, online scoring, and online interpretation. Since its launch, it has accumulated a large amount of user and rural data nationwide. This image and text data contains latitude and longitude information, enabling precise location tracking and providing a new means of observing the rural living environment. "Village Scenery Photo Shooting" provides a large number of location-coded rural images, which can serve as an effective source of rural geographic points of interest (POIs).

[0003] A Chinese invention patent published on November 2, 2021, entitled "A Method for Classifying Urban Landscape Elements and Measuring Locality Based on Street View Images," firstly classifies urban street view image data into 365 subcategories and constructs a classification system for local landscape elements to reclassify the scenes. Secondly, it measures the locality level of each city from two perspectives: locality value and similarity value, according to this classification system. Finally, it couples the classified landscape elements with spatial elements such as road accessibility, facility density, and functional diversity for analysis, characterizing the change process and mechanism of urban locality to determine the types of landscape elements influencing local differences and similarities. However, similarly, the above data is mainly distributed in urban areas, and its geographic label classification system is also mainly concentrated on urban feature scenes. Therefore, the existing technology still has certain limitations. Summary of the Invention

[0004] To address the limitations of existing technologies, this invention proposes a method for updating rural geographic tags based on social media image data. The technical solution adopted by this invention is as follows:

[0005] A method for updating rural geotags based on social media image data includes the following steps:

[0006] S1, Obtain images containing rural geography content collected from social media for prediction;

[0007] S2, input the image to be predicted into an object detection network trained on a preset object detection dataset, detect the rural facilities contained in the image to be predicted, and obtain the classification result of the image to be predicted regarding the rural facility scene;

[0008] S3, input the image to be predicted into a multi-label classification network trained on a preset multi-label classification dataset to obtain the classification result of the image to be predicted regarding the rural environment scene;

[0009] S4. The classification results of steps S2 and S3 are merged to complete the rural geographic label update of the image to be predicted.

[0010] Compared with existing technologies, this invention proposes a rural geotag update method based on social media image data. By combining multiple deep learning methods, it improves prediction accuracy while ensuring prediction efficiency. It can automatically update image tags and classify them for various different scenarios, making it widely applicable, effectively improving the efficiency of rural geotag updates, and reducing labor costs and workload.

[0011] As a preferred embodiment, the image data of the target detection dataset is labeled with the following data tags for rural facilities: traditional buildings, farmhouses, garbage collection points, schools, public toilets, village committees, and village entrances;

[0012] Furthermore, in the target detection network, the category threshold for village entrance is 0.2, the category threshold for garbage collection point is 0.35, and the category threshold for traditional buildings, farmhouses, schools, public toilets, and village committees is 0.25.

[0013] Furthermore, in step S2, after detecting rural facilities contained in the image to be predicted, the mutually exclusive cases in the classification results are post-processed in the following manner:

[0014] S21, determine whether the category of the classification result includes farmhouses or traditional buildings: otherwise, do not process; if yes, determine whether the ratio of the area of ​​the detection box of farmhouses or traditional buildings to the area of ​​the image to be predicted is less than 5%; if yes, discard the detection box and the corresponding category; otherwise, proceed to step S22.

[0015] S22, determine whether the category of the classification result contains both farmhouses and village committees or both traditional buildings and village committees. Otherwise, no processing is performed. If yes, determine whether the detection box of the village committee is located inside the detection box of farmhouses or traditional buildings. If yes, discard the farmhouse category or traditional building category and the corresponding location information contained in the classification result. Otherwise, no processing is performed.

[0016] As a preferred embodiment, the image data of the multi-label classification dataset is labeled with the following data labels for rural environments: roads, farmland, water bodies, and small squares.

[0017] Furthermore, the multi-label classification dataset is preprocessed as follows before being used for training:

[0018] The image data of the multi-label classification dataset is enhanced by adding Gaussian noise, increasing brightness, decreasing brightness, increasing contrast, and decreasing contrast.

[0019] The data features of the image data in the multi-label classification dataset are normalized so that the values ​​of the image matrix are between 0 and 1;

[0020] Convert the data labels of the image data in the multi-label classification dataset into n-hot encoding format (1 <= n <= 4).

[0021] As a preferred embodiment, the target detection network is the YOLO-v5 target detection network;

[0022] The multi-label classification network is a ResNet50 multi-label classification network. The last layer of the multi-label classification network uses the sigmoid activation function, and the optimizer of the multi-label classification network is the Adam optimizer.

[0023] This invention also includes the following:

[0024] A rural geotag update system based on social media image data includes a module for acquiring images to be predicted, a module for classifying rural facilities and scenes, a module for classifying rural environment and scenes, and a module for merging classification results. The module for acquiring images to be predicted is connected to both the module for classifying rural facilities and scenes and the module for classifying rural environment and scenes. The module for merging classification results is connected to both the module for classifying rural facilities and scenes and the module for classifying rural environment and scenes.

[0025] The image acquisition module is used to acquire images containing rural geography content collected from social media.

[0026] The rural facility scene classification module is used to input the image to be predicted into a target detection network trained on a preset target detection dataset, detect the rural facilities contained in the image to be predicted, and obtain the classification result of the image to be predicted regarding the rural facility scene.

[0027] The rural environment scene classification module is used to input the image to be predicted into a multi-label classification network trained on a preset multi-label classification dataset to obtain the classification result of the image to be predicted regarding the rural environment scene.

[0028] The classification result merging module is used to merge the classification results of the rural facility scene classification module and the rural environment scene classification module to complete the rural geographic label update of the image to be predicted.

[0029] Compared to existing technologies, this invention proposes a rural geotag update system based on social media image data. By combining multiple deep learning methods, it improves prediction accuracy while ensuring prediction efficiency. It can automatically update image tags and classify them for various different scenarios, making it widely applicable, effectively improving the efficiency of rural geotag updates, and reducing labor costs and workload.

[0030] A storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned method for updating rural geographic tags based on social media image data.

[0031] A computer device includes a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the aforementioned method for updating rural geotags based on social media image data. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating a method for updating rural geographic tags based on social media image data, as provided in Embodiment 1 of the present invention.

[0033] Figure 2 Legend for the target detection dataset of this invention;

[0034] Figure 3 This is an example of a mutually exclusive case in the classification results of this invention;

[0035] Figure 4 This is an illustration of the multi-label classification dataset of the present invention;

[0036] Figure 5 This is a schematic diagram of a rural geographic tag update system based on social media image data provided in Embodiment 2 of the present invention. Detailed Implementation

[0037] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0038] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0039] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0040] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0041] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] To address the limitations of existing technologies, this embodiment provides a technical solution. The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] Example 1

[0044] For a method of updating rural geotags based on social media image data, please refer to [link / reference]. Figure 1 This includes the following steps:

[0045] S1, Obtain images containing rural geography content collected from social media for prediction;

[0046] S2, input the image to be predicted into an object detection network trained on a preset object detection dataset, detect the rural facilities contained in the image to be predicted, and obtain the classification result of the image to be predicted regarding the rural facility scene;

[0047] S3, input the image to be predicted into a multi-label classification network trained on a preset multi-label classification dataset to obtain the classification result of the image to be predicted regarding the rural environment scene;

[0048] S4. The classification results of steps S2 and S3 are merged to complete the rural geographic label update of the image to be predicted.

[0049] Compared with existing technologies, this invention proposes a rural geotag update method based on social media image data. By combining multiple deep learning methods, it improves prediction accuracy while ensuring prediction efficiency. It can automatically update image tags and classify them for various different scenarios, making it widely applicable, effectively improving the efficiency of rural geotag updates, and reducing labor costs and workload.

[0050] There is no strict order between steps S2 and S3; they can be performed simultaneously.

[0051] As a preferred embodiment, please refer to Figure 2 The image data in the target detection dataset is labeled with the following data tags for rural facilities: traditional buildings, farmhouses, garbage collection points, schools, public toilets, village committees, and village entrances.

[0052] Furthermore, in the target detection network, the category threshold for village entrance is 0.2, the category threshold for garbage collection point is 0.35, and the category threshold for traditional buildings, farmhouses, schools, public toilets, and village committees is 0.25.

[0053] Specifically, in the data tags for rural facilities in this embodiment, traditional buildings refer to ancestral halls, temples, and other buildings with traditional architectural styles; farmhouses refer to farmhouses in the foreground of the image, excluding village committee buildings, school buildings, and public toilets; garbage collection points mainly include garbage bins and semi-exposed garbage collection points; schools include kindergartens, primary schools, and middle schools, and the images contain the words "primary school," "middle school," "kindergarten," and "school"; images of public toilets contain the words "men's," "women's," or "toilet," or signs indicating men's and women's restrooms; village committees include public service venues such as committees, health stations, service stations, and service centers, and the images should contain the words "committee," "health station," "service station," or "service center," or corresponding signs, as well as the village name; images of the village entrance contain the village name, which can be located on objects such as stones or signs, and the village name must be clear and unobstructed.

[0054] The object detection dataset specifically includes image data and corresponding annotation data. That is, in the image, each identifiable object (or a line of text, an icon) is an instance, and the annotation information corresponding to each instance includes: the category of the instance, and the position information of the smallest matrix surrounding the instance.

[0055] Furthermore, in step S2, after detecting rural facilities contained in the image to be predicted, the mutually exclusive cases in the classification results are post-processed in the following manner:

[0056] S21, determine whether the category of the classification result includes farmhouses or traditional buildings: otherwise, do not process; if yes, determine whether the ratio of the area of ​​the detection box of farmhouses or traditional buildings to the area of ​​the image to be predicted is less than 5%; if yes, discard the detection box and the corresponding category; otherwise, proceed to step S22.

[0057] S22, determine whether the category of the classification result contains both farmhouses and village committees or both traditional buildings and village committees. Otherwise, no processing is performed. If yes, determine whether the detection box of the village committee is located inside the detection box of farmhouses or traditional buildings. If yes, discard the farmhouse category or traditional building category and the corresponding location information contained in the classification result. Otherwise, no processing is performed.

[0058] Specifically, with Figure 3 For example, the classification result of this image contains both farmhouses and village committees, and the detection box of the village committee is located inside the detection box of the farmhouse. After step S21, the farmhouse and corresponding location information of the image will be discarded, and the final classification result of the image will be retained as village committee.

[0059] As a preferred embodiment, please refer to Figure 4 The image data in the multi-label classification dataset is labeled with the following data labels for rural environments: roads, farmland, water bodies, and small squares.

[0060] Specifically, in the data tags for the rural environment in this embodiment, roads include various types of rural roads, such as asphalt roads, cement roads, and dirt roads; farmland includes obvious and large areas for producing crops; water bodies mainly include ponds and rivers; and small squares refer to rural activity squares, which include fitness equipment, basketball hoops, slogans, sculptures, etc.

[0061] Furthermore, the multi-label classification dataset is preprocessed as follows before being used for training:

[0062] The image data of the multi-label classification dataset is enhanced by adding Gaussian noise, increasing brightness, decreasing brightness, increasing contrast, and decreasing contrast.

[0063] The data features of the image data in the multi-label classification dataset are normalized so that the values ​​of the image matrix are between 0 and 1;

[0064] Convert the data labels of the image data in the multi-label classification dataset into n-hot encoding format (1 <= n <= 4).

[0065] In a preferred embodiment, the target detection network is the YOLO-v5 target detection network;

[0066] The multi-label classification network is a ResNet50 multi-label classification network. The last layer of the multi-label classification network uses the sigmoid activation function, and the optimizer of the multi-label classification network is the Adam optimizer.

[0067] Specifically, the YOLOv5 detection model is used as the network framework for detecting seven categories, which can detect targets in the image in real time while ensuring detection accuracy. The image category is obtained through post-processing. The ResNet50 classification model is used as the network framework for classifying four categories: roads, squares, farmland, and water bodies. Finally, the classification results of the two models are merged to obtain the final classification result of the image, realizing real-time image classification.

[0068] The learning rate (lr) of the multi-label classification network is initially set to 1e-3, and will be dynamically adjusted during training based on the changes in the full readiness rate of the preset validation set.

[0069] During the training of the multi-label classification network, thresholds for each category need to be selected simultaneously, and the validation set complete accuracy is obtained based on the model and thresholds. Training continues until a given epoch is reached. When the given epoch is reached, the model and threshold combination with the highest validation set complete accuracy is output. Finally, the image to be predicted is input into the model and threshold combination with the highest validation set complete accuracy for image classification.

[0070] Example 2

[0071] Please see Figure 5 A rural geotag update system based on social media image data includes a predictable image acquisition module 1, a rural facility scene classification module 2, a rural environment scene classification module 3, and a classification result merging module 4; the predictable image acquisition module 1 is connected to the rural facility scene classification module 2 and the rural environment scene classification module 3 respectively; the classification result merging module 4 is connected to the rural facility scene classification module 2 and the rural environment scene classification module 3 respectively; wherein:

[0072] The image acquisition module 1 is used to acquire images containing rural geography content collected from social media.

[0073] The rural facility scene classification module 2 is used to input the image to be predicted into a target detection network trained on a preset target detection dataset, detect the rural facilities contained in the image to be predicted, and obtain the classification result of the image to be predicted regarding the rural facility scene.

[0074] The rural environment scene classification module 3 is used to input the image to be predicted into a multi-label classification network trained on a preset multi-label classification dataset to obtain the classification result of the image to be predicted regarding the rural environment scene.

[0075] The classification result merging module 4 is used to merge the classification results of the rural facility scene classification module 2 and the rural environment scene classification module 3 to complete the rural geographic label update of the image to be predicted.

[0076] Compared to existing technologies, this invention proposes a rural geotag update system based on social media image data. By combining multiple deep learning methods, it improves prediction accuracy while ensuring prediction efficiency. It can automatically update image tags and classify them for various different scenarios, making it widely applicable, effectively improving the efficiency of rural geotag updates, and reducing labor costs and workload.

[0077] Example 3

[0078] A storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the rural geographic tag update method based on social media image data as described in Example 1.

[0079] Example 4

[0080] A computer device includes a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the rural geotag update method based on social media image data as described in Embodiment 1.

[0081] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for updating rural geographic tags based on social media image data, characterized in that, Includes the following steps: S1, Obtain images containing rural geography content collected from social media for prediction; S2, input the image to be predicted into an object detection network trained on a preset object detection dataset, detect the rural facilities contained in the image to be predicted, and obtain the classification result of the image to be predicted regarding the rural facility scene; After detecting rural facilities in the image to be predicted, the mutually exclusive cases in the classification results are post-processed in the following way: S21, determine whether the category of the classification result includes farmhouses or traditional buildings: otherwise, do not process; if yes, determine whether the ratio of the area of ​​the detection box of farmhouses or traditional buildings to the area of ​​the image to be predicted is less than 5%; if yes, discard the detection box and the corresponding category; otherwise, proceed to step S22. S22, determine whether the category of the classification result contains both farmhouses and village committees or both traditional buildings and village committees. Otherwise, no processing is performed. If yes, determine whether the detection box of the village committee is located inside the detection box of farmhouses or traditional buildings. If yes, discard the farmhouse category or traditional building category and the corresponding location information contained in the classification result. Otherwise, no processing is performed. S3, input the image to be predicted into a multi-label classification network trained on a preset multi-label classification dataset to obtain the classification result of the image to be predicted regarding the rural environment scene; S4. The classification results of steps S2 and S3 are merged to complete the rural geographic label update of the image to be predicted.

2. The method for updating rural geographic tags based on social media image data according to claim 1, characterized in that, The image data in the target detection dataset is labeled with the following data tags for rural facilities: traditional buildings, farmhouses, garbage collection points, schools, public toilets, village committees, and village entrances.

3. The method for updating rural geographic tags based on social media image data according to claim 2, characterized in that, In the target detection network, the category threshold for village entrance is 0.2, the category threshold for garbage collection point is 0.35, and the category threshold for traditional buildings, farmhouses, schools, public toilets, and village committees is 0.

25.

4. The method for updating rural geographic tags based on social media image data according to claim 1, characterized in that, The image data in the multi-label classification dataset is labeled with the following rural environment data labels: roads, farmland, water bodies, and small squares.

5. The method for updating rural geographic tags based on social media image data according to claim 4, characterized in that, Before being used for training, the multi-label classification dataset undergoes the following preprocessing: The image data of the multi-label classification dataset is enhanced by adding Gaussian noise, increasing brightness, decreasing brightness, increasing contrast, and decreasing contrast. The data features of the image data in the multi-label classification dataset are normalized so that the values ​​of the image matrix are between 0 and 1; Convert the data labels of the image data in the multi-label classification dataset into n-hot encoding format (1 <= n <= 4).

6. The method for updating rural geographic tags based on social media image data according to claim 1, characterized in that, The target detection network is the YOLO-v5 target detection network; The multi-label classification network is a ResNet50 multi-label classification network. The last layer of the multi-label classification network uses the sigmoid activation function, and the optimizer of the multi-label classification network is the Adam optimizer.

7. A rural geotag update system based on social media image data, characterized in that, It includes a module for acquiring images to be predicted (1), a module for classifying rural facilities scenes (2), a module for classifying rural environment scenes (3), and a module for merging classification results (4); the module for acquiring images to be predicted (1) is connected to the module for classifying rural facilities scenes (2) and the module for classifying rural environment scenes (3); the module for merging classification results (4) is connected to the module for classifying rural facilities scenes (2) and the module for classifying rural environment scenes (3); wherein: The image acquisition module (1) is used to acquire images containing rural geography content collected from social media. The rural facility scene classification module (2) is used to input the image to be predicted into a target detection network trained on a preset target detection dataset, detect the rural facilities contained in the image to be predicted, and obtain the classification result of the image to be predicted regarding the rural facility scene. After detecting rural facilities in the image to be predicted, the mutually exclusive cases in the classification results are post-processed in the following way: S21, determine whether the category of the classification result includes farmhouses or traditional buildings: otherwise, do not process; if yes, determine whether the ratio of the area of ​​the detection box of farmhouses or traditional buildings to the area of ​​the image to be predicted is less than 5%; if yes, discard the detection box and the corresponding category; otherwise, proceed to step S22. S22, determine whether the category of the classification result contains both farmhouses and village committees or both traditional buildings and village committees. Otherwise, no processing is performed. If yes, determine whether the detection box of the village committee is located inside the detection box of farmhouses or traditional buildings. If yes, discard the farmhouse category or traditional building category and the corresponding location information contained in the classification result. Otherwise, no processing is performed. The rural environment scene classification module (3) is used to input the image to be predicted into a multi-label classification network trained on a preset multi-label classification dataset to obtain the classification result of the image to be predicted regarding the rural environment scene; The classification result merging module (4) is used to merge the classification results of the rural facility scene classification module (2) and the rural environment scene classification module (3) to complete the rural geographic label update of the image to be predicted.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the rural geographic tag update method based on social media image data as described in any one of claims 1 to 6.

9. A computer device, characterized in that: The method includes a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the rural geotag update method based on social media image data as described in any one of claims 1 to 6.

Images