A method and system for building a city infrastructure platform

Abstract

The application relates to the technical field of smart cities, in particular to a method and system for building a city infrastructure platform, comprising: performing annotation processing on acquired city multi-modal data; constructing a question and answer sample set, a scene pairing data set and a preference data set based on the city multi-modal data after annotation processing; after preliminary training of a model deployed in a model warehouse based on the question and answer sample set and the scene pairing data set, fine-tuning the deployed model based on the preference data set; based on the city multi-modal data after annotation processing, constructing a structured database, an unstructured database and a knowledge graph; based on a pre-set API interface, performing API service building on a city scene task support model and a language large model after training, and the structured database, the unstructured database and the knowledge graph, so as to obtain a built city infrastructure platform. The application effectively improves the intelligent level and service efficiency of city governance.

Description

Technical Field

[0001] This application relates to the field of smart city technology, and in particular to a method and system for building an urban infrastructure platform. Background Technology

[0002] With the acceleration of urbanization and the deep development of digital technology, the construction of smart cities has become an important direction for improving urban governance capabilities and optimizing the quality of public services. A smart city aims to achieve intelligent management of multiple scenarios, including urban transportation, municipal facilities, and public safety. The realization of a smart city highly relies on the efficient integration and application of multimodal urban data. This means that the current operation of a city generates massive amounts of text, images, videos, and audio data, which are the technologies for perceiving the city's status and optimizing governance decisions.

[0003] In existing smart city management platforms, on the one hand, they rely heavily on single models to process text, images, or time-series data, making it difficult to integrate multi-source heterogeneous data (such as analyzing the correlation between citizen complaint texts and camera images). This results in the need for manual intervention in complex scenarios (such as identifying street vendors), leading to low efficiency in handling such cases. On the other hand, they rely heavily on single-function models (such as independent visual detection models and text processing models), lacking collaborative capabilities between models. Furthermore, they require repeated data collection and model training for different scenarios, resulting in long smart scenario delivery cycles (usually monthly) and insufficient adaptability and scalability. Summary of the Invention

[0004] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the prior art, this application provides a method and system for building an urban infrastructure platform, which solves the technical problems of existing smart city management platforms, which rely on a single model to process text, image or time series data, making it difficult to integrate multi-source heterogeneous data, resulting in the need for manual intervention in complex scenarios, and rely on a single functional model, requiring repeated data collection and model training for different scenarios, resulting in long smart scenario delivery cycles and insufficient adaptability and scalability.

[0005] (II) Technical Solution To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a method for constructing an urban infrastructure platform, comprising: Acquire urban multimodal data and perform annotation processing on the urban multimodal data; Based on the annotated urban multimodal data, a question-answering sample set, a scene matching dataset, and a preference dataset are constructed. The question-answering sample set is a dataset constructed using a prompt word generation algorithm, the scene matching dataset is a dataset constructed using a multimodal data association algorithm, and the preference dataset is a dataset constructed using a pre-set priority strategy. The urban scene task support model pre-deployed in the model repository is initially trained based on the scene matching dataset, and the language large model pre-deployed in the model repository is initially trained based on the question answering sample set. The urban scene task support model and the language large model are fine-tuned based on the preference dataset to obtain the trained urban scene task support model and language large model. Based on the annotated urban multimodal data, a structured database and an unstructured database are constructed, and an NLP algorithm is used to perform entity recognition, relationship processing and attribute normalization on the urban multimodal data in order to construct a knowledge graph. Based on pre-set API interfaces, API services are built for the trained urban scene task support model and language large model, as well as structured databases, unstructured databases and knowledge graphs, to obtain a completed urban infrastructure platform.

[0006] Optionally, in one specific embodiment, acquiring urban multimodal data and labeling the urban multimodal data includes: Acquire urban multimodal data and preprocess the urban multimodal data based on the data types, including text, image, video, and audio. The preprocessed urban multimodal data is then labeled. The preprocessing includes: The invalid information of the text-type urban multimodal data is removed based on the keyword matching algorithm, and the text-type urban multimodal data is deduplicated based on the pre-set hash algorithm. Then, the deduplicated text-type urban multimodal data is cleaned for noise, text standardization and format conversion. Invalid information is removed from the image-type urban multimodal data based on the image quality assessment algorithm. After denoising the image-type urban multimodal data using median filtering and Gaussian filtering algorithms, the image-type urban multimodal data is normalized and target recognition is performed. Invalid data in urban multimodal data of video type is removed based on scene recognition algorithm, and frame extraction is performed on urban multimodal data of video type to perform noise reduction, size standardization and target localization on the extracted image; The speech quality assessment algorithm is used to evaluate the quality of urban multimodal data of speech type. After removing noise from the speech type urban multimodal data using the Wiener filtering algorithm, the speech type urban multimodal data is standardized and processed for speech-to-text conversion.

[0007] Optionally, in one specific embodiment, the urban multimodal data includes urban data of multiple different data types; wherein, the data types include text type, image type, video type, and voice type; Based on annotated urban multimodal data, a question-answering sample set, a scene matching dataset, and a preference dataset were constructed, including: Based on the data type and label of each city in the annotated multimodal data of cities, all city data are classified to divide each city data into text-dominant data, multimodal correlated data and user feedback correlated data, and the same city data can be one or more of the following: text-dominant data, multimodal correlated data and user feedback correlated data. Based on a pre-set prompt word generation tool, question-answer pairs are constructed for each text-dominant data to build a question-answer sample set; Based on the data type and label corresponding to each multimodal association data, multimodal association data with the same scene label but belonging to different data types are paired to construct a scene pairing dataset. The scene label is obtained by feature extraction from the label of the multimodal association data. A preference dataset is constructed based on the data type and labeling of each user's feedback data.

[0008] Optionally, in a specific embodiment, the urban scene task support model and the large language model, which have been initially trained, are fine-tuned based on the preference dataset, including: Based on the preference dataset, the urban scene task support model and the language large model after initial training were subjected to structured pruning. Furthermore, based on the preference dataset, the structured pruning urban scene task support model and the language big model were fine-tuned respectively.

[0009] Optionally, in one specific embodiment, the structured pruning process includes: Identify the fully connected layers in the urban scene task support model or language large model that require pruning; Based on the pre-set low-rank dimension, the first low-rank matrix and the second low-rank matrix are inserted into the fully connected layer that needs to be pruned in order to update the urban scene task support model or language large model. The dimension of the first low-rank matrix is ​​the input feature dimension × low-rank dimension, and the dimension of the second low-rank matrix is ​​the low-rank dimension × output feature dimension. The preference dataset is input into the updated urban scene task support model or language large model for training. The first update gradient corresponding to the first low-rank matrix and the first update gradient corresponding to the second low-rank matrix are obtained through the backpropagation algorithm. Based on the first update gradient corresponding to the first low-rank matrix and the first update gradient corresponding to the second low-rank matrix, some model parameters in the urban scene task support model or language large model are removed.

[0010] Optionally, in a specific embodiment, updating the urban scene task support model or the large language model includes: For the fully connected layer in the urban scene task support model or language large model, the input-output relationship of the fully connected layer is reconstructed based on the first low-rank matrix, the second low-rank matrix, and a pre-set formula 1; the formula 1 is: Y = X·w0 + A·B; Where Y is the output of the fully connected layer, X is the input of the fully connected layer, w0 is the pre-set initial weight, A is the first low-rank matrix, and B is the second low-rank matrix.

[0011] Optionally, in a specific embodiment, when removing some model parameters, the model parameters in the attention layer and FFN layer of some models in the urban scene task support model or the language big model are grouped, and the importance of all model parameters in each group is weighted and summed to obtain the corresponding group importance, so as to remove model parameters whose group importance is lower than the preset importance threshold.

[0012] Optionally, in a specific embodiment, the structured pruned urban scene task support model and the large language model are fine-tuned separately based on the preference dataset, including: The structured and pruned urban scene task support model or language large model is used as the initial policy network, and the reward function corresponding to the city data is constructed based on the label corresponding to each city data in the preference dataset. Data from any city in the preference dataset is used as input to the initial policy network to generate multiple candidate outputs; Based on the similarity between the reward function corresponding to the city data input to the initial policy network and each candidate output, and using a pre-set formula two, the reward parameter corresponding to each candidate output is obtained; the formula two is: ; Where R is the reward parameter and P is the similarity between the candidate output and the reward function; Based on a pre-set preference optimization algorithm and reward parameters, the first and second low-rank matrices in the initial policy network are optimized for model fine-tuning.

[0013] Optionally, in a specific embodiment, based on a pre-set preference optimization algorithm and reward parameters, the first low-rank matrix and the second low-rank matrix in the initial policy network are optimized for model fine-tuning, including: Based on the pre-set preference optimization algorithm and reward parameters, the corresponding target loss function is obtained, and the second update gradient of the target loss function with respect to the first low-rank matrix and the second low-rank matrix is ​​obtained through the backpropagation algorithm. The first low-rank matrix and the second gradient matrix are updated based on the second update gradient of the first low-rank matrix and the second low-rank matrix, and the AdamW optimization algorithm is used to fine-tune the model.

[0014] Secondly, embodiments of this application provide a system for building an urban infrastructure platform, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the above-described method for building an urban infrastructure platform.

[0015] (III) Beneficial Effects This application discloses a method for building an urban infrastructure platform. It comprehensively collects and labels multimodal urban data to construct a high-quality, structured data foundation. Based on this foundation, it combines prompt word generation algorithms, multimodal association algorithms, and priority strategies to construct question-answering sample sets, scene matching datasets, and preference datasets, achieving data adaptation to urban scenarios and model tasks. The method then uses these two datasets to initially train an urban scenario task support model and a large language model, respectively. Finally, it fine-tunes the model using the preference dataset, ensuring that the model's capabilities align with both urban business needs and practical application preferences. By constructing structured databases, unstructured databases, and knowledge graphs, it integrates multiple types of data resources and structured knowledge, providing robust data support for the platform. Finally, it uses standardized API interfaces to build services for the model, database, and knowledge graph, forming a fully functional, collaborative, and efficient urban infrastructure platform that effectively improves the intelligence level and service efficiency of urban governance. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a method for constructing an urban infrastructure platform, as provided in an embodiment of this application; Figure 2 This is a schematic diagram of the urban infrastructure platform structure provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of building an urban infrastructure platform as provided in this application embodiment. Detailed Implementation

[0017] To better explain and facilitate understanding of this application, the following detailed description of the application is provided in conjunction with the accompanying drawings and specific embodiments.

[0018] With the deep integration of urbanization and digital technology, smart city construction has become a core direction for improving urban governance efficiency and optimizing public services. Smart cities need to achieve intelligent management across multiple scenarios such as transportation, municipal administration, and security. This goal highly depends on the efficient integration and application of multimodal data such as text, images, videos, and voice generated during urban operations. However, existing smart city management platforms have significant shortcomings: they often rely on single models to process specific types of data, making it difficult to integrate multi-source heterogeneous data; complex scenarios require manual intervention, leading to low efficiency; and there is a lack of collaboration between models, requiring repeated data collection and model training, resulting in long smart scenario delivery cycles and insufficient adaptability and scalability, thus hindering the advancement of smart city construction.

[0019] This application provides a method for building an urban infrastructure platform. By comprehensively collecting and labeling urban multimodal data, a high-quality structured data foundation is established. Combining prompt word generation algorithms, multimodal association algorithms, and priority strategies, three types of datasets adapted to urban scenarios and model tasks are constructed. These datasets are used for the initial training and preference fine-tuning of urban scenario task support models and large language models, respectively, ensuring that model capabilities both meet urban business needs and adapt to practical application preferences. Simultaneously, structured and unstructured databases and knowledge graphs are built, integrating multiple types of data resources and structured knowledge. Service construction is then completed through standardized API interfaces, forming a fully functional and highly efficient platform. This effectively improves the level of intelligent urban governance and service efficiency, providing strong support for the implementation of smart cities.

[0020] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0021] This application provides a method for constructing an urban infrastructure platform, such as... Figure 1 As shown, it includes: S1. Acquire urban multimodal data and perform annotation processing on the urban multimodal data; S2. Construct a question-answering sample set, a scene matching dataset, and a preference dataset based on the labeled urban multimodal data. The question-answering sample set is a dataset constructed using a prompt word generation algorithm, the scene matching dataset is a dataset constructed using a multimodal data association algorithm, and the preference dataset is a dataset constructed using a pre-set priority strategy. S3. Based on the scene matching dataset, the pre-deployed urban scene task support model in the model repository is initially trained, and the pre-deployed language large model in the model repository is initially trained based on the question answering sample set. Based on the preference dataset, the pre-trained urban scene task support model and language large model are fine-tuned to obtain the trained urban scene task support model and language large model. S4. Based on the labeled urban multimodal data, construct a structured database and an unstructured database, and use NLP algorithms to perform entity recognition, relationship processing and attribute normalization on the urban multimodal data in order to construct a knowledge graph. S5. Based on the pre-set API interface, build API services for the trained urban scene task support model and language big model, as well as structured databases, unstructured databases and knowledge graphs, to obtain the completed urban infrastructure platform.

[0022] This application provides a method for building an urban infrastructure platform. By comprehensively collecting and labeling urban multimodal data, a high-quality structured data foundation is established. Combining prompt word generation algorithms, multimodal association algorithms, and priority strategies, three types of datasets adapted to urban scenarios and model tasks are constructed. These datasets are used for the initial training and preference fine-tuning of urban scenario task support models and large language models, respectively, ensuring that model capabilities both meet urban business needs and adapt to practical application preferences. Simultaneously, structured and unstructured databases and knowledge graphs are built, integrating multiple types of data resources and structured knowledge. Service construction is then completed through standardized API interfaces, forming a fully functional and highly efficient platform. This effectively improves the intelligence level and service efficiency of urban governance, providing strong support for the implementation of smart cities.

[0023] Optionally, in one specific embodiment, acquiring urban multimodal data and labeling the urban multimodal data includes: Acquire urban multimodal data and preprocess the urban multimodal data based on the data types, including text, image, video, and audio. The preprocessed urban multimodal data is then labeled. Preprocessing includes: The invalid information of the text-type urban multimodal data is removed based on the keyword matching algorithm, and the text-type urban multimodal data is deduplicated based on the pre-set hash algorithm. Then, the deduplicated text-type urban multimodal data is cleaned for noise, text standardization and format conversion. Invalid information is removed from the image-type urban multimodal data based on the image quality assessment algorithm. After denoising the image-type urban multimodal data using median filtering and Gaussian filtering algorithms, the image-type urban multimodal data is normalized and target recognition is performed. Invalid data in urban multimodal data of video type is removed based on scene recognition algorithm, and frame extraction is performed on urban multimodal data of video type to perform noise reduction, size standardization and target localization on the extracted image; The speech quality assessment algorithm is used to evaluate the quality of urban multimodal data of speech type. After removing noise from the speech type urban multimodal data using the Wiener filtering algorithm, the speech type urban multimodal data is standardized and processed for speech-to-text conversion.

[0024] Specifically, in the context of building smart city infrastructure platforms, urban multimodal data refers to a heterogeneous data collection covering all aspects of urban operation, encompassing various forms. It is primarily categorized into four basic types: text, image, video, and audio. In some scenarios, this can extend to time-series sensor data, spatial geographic data, and more. These data acquisition channels are deeply integrated with urban governance business systems and terminal devices.

[0025] Specific examples of how the above data was obtained are provided below: Text data types include, but are not limited to: government affairs data (policy documents, enforcement procedures, work order records, inter-departmental collaboration documents, etc.), citizen data (complaint texts, consultation messages, government service application information, etc.), and operational data (traffic flow statistics reports, facility maintenance logs, incident handling reports, etc.). Acquisition methods typically involve direct integration with various departmental systems, supplemented by manual data entry. This involves directly accessing the business system APIs of various city departments to synchronize structured text data. For example, accessing the municipal management system to obtain manhole cover repair work orders (with fields such as fault location, reporting time, and responsible department); accessing the traffic management system to obtain traffic violation penalty decisions; and accessing the government service platform to obtain residents' online application information and feedback texts. Alternatively, it can be collected through official apps, mini-programs, public accounts, service websites, etc., to gather resident complaints and consultation texts. Alternatively, paper complaint letters and suggestion books can be converted into electronic documents via OCR scanning (manual / automatic) before data entry. Or, based on compliance agreements, publicly available data such as official documents and announcements can be crawled.

[0026] Image data types include, but are not limited to: surveillance data (road surveillance snapshots, subway passenger flow snapshots, photos of manhole covers / streetlights and other facility inspections, etc.), aerial data (drone images of illegal construction investigations, urban green coverage maps), and reported data (photos taken by citizens of vehicles illegally occupying roads, photos of malfunctioning or illegal facilities / buildings, etc.). Image data types are typically acquired through terminal device capture or manual reporting, with automatic device acquisition being the primary method and manual reporting as a supplement. For example, high-definition cameras are deployed in key areas such as urban roads, subway stations, commercial districts, and parks, and capture trigger rules are set to automatically collect images; or, for large-scale, hard-to-reach areas (such as urban elevated roads, old residential areas, and riverbanks), drones are used to take aerial photos along preset routes to obtain image data of scenarios such as illegal construction investigations, green coverage, and river garbage; or the "take a picture" function of government affairs apps encourages citizens to take and upload images of violations and facility malfunctions; law enforcement officers and inspectors take on-site images using handheld terminals and upload them in real time.

[0027] Video data types include, but are not limited to: real-time surveillance videos (intersection traffic monitoring videos, shopping mall pedestrian monitoring videos, subway car monitoring videos, etc.) and specialized videos (law enforcement process recording videos, facility repair process videos, emergency drill videos, etc.). Video data is typically acquired through real-time monitoring streams and specialized recordings, primarily through real-time streaming and batch import. Examples include: connecting to a city's dedicated video network to acquire real-time video streams from monitoring equipment deployed by various departments; law enforcement officers recording videos using body cameras during law enforcement operations; facility maintenance personnel recording construction process videos at repair sites; and emergency management departments recording videos at emergency drills and accident response sites, which are then imported in batches afterward.

[0028] Voice data types include, but are not limited to: hotline records (12345 video complaint recordings, government consultation call recordings, etc.), broadcasts (traffic radio broadcasts, emergency command and dispatch voice recordings, etc.), and on-site recordings (law enforcement personnel on-site recordings, residents on-site recordings, etc.). Voice data is typically collected through recording equipment and transcribed in real-time. For example: connecting to the 12345 citizen hotline system automatically records citizen complaints and inquiries, while simultaneously using ASR (Automatic Speech Recognition) technology to convert the speech to text in real-time, storing two copies of the data (voice file + text file); or, law enforcement and inspection personnel are equipped with handheld terminals with recording capabilities to record voice during on-site inquiries and communications; real-time recording of dispatch calls in the emergency command center documents the command and decision-making process; or, accessing voice streams from traffic radio and emergency broadcast systems to obtain voice data such as traffic condition broadcasts and disaster warning broadcasts.

[0029] Further develop standardized data access interfaces (such as RESTful API, gRPC, etc.) to ensure seamless data integration with the platform from different departments and devices; establish a tiered access control system for sensitive data, allowing authorized personnel to view and access it, thus ensuring data security; and set collection frequencies according to data importance, such as real-time access to traffic video streams, scheduled synchronization of policy documents, and immediate collection of video feedback.

[0030] Examples of specific preprocessing methods are provided for four types of data: Text-type data (video complaint work order), such as: "[Invalid information] The manhole cover on XX Road is broken and no one is fixing it! The address is roughly near the east gate of XX Community. It broke last week and no one has come to fix it yet."

[0031] First, invalid information is removed. Based on a keyword matching algorithm (with a pre-set database of invalid keywords such as "spam" and "advertisement"), the beginning of "spam" is removed and the garbled characters "####" are marked. Secondly, perform hash deduplication by calculating the text hash value. If the complaint ticket already exists (e.g., the same resident submits it repeatedly), only one copy will be retained. Furthermore, noise cleanup and standardization: remove honorary exclamation marks, correct colloquial descriptions, standardize address formats, and finally obtain "The manhole cover near the east gate of XX Community on XX Road is damaged. It was discovered last week and poses a safety hazard to pedestrians." Then, the format is converted from text to JSON, with fields including "content, source, time, and preliminary category".

[0032] After preprocessing, city scene labels are generated using automatic labeling tools or manually. Labeling information: Problem type: Facility malfunction (manhole cover); Location: East gate of XX Community, XX Road; Urgency level: Medium; Responsible department: Municipal maintenance department.

[0033] Image-type data, such as (road surveillance capture images): Images captured by surveillance cameras at XX intersection (with backlighting and blurring, a small amount of rain and snow noise, and the image contains vehicles operating on the road, pedestrians, and background buildings).

[0034] First, invalid information is removed: an image quality assessment algorithm is used (to calculate sharpness and signal-to-noise ratio). If the image sharpness is ≥ a threshold (e.g., 30dB), it is considered valid data; if it is completely black or so blurry that the target cannot be identified, it is directly removed. Subsequently, noise reduction processing was performed: first, median filtering was used to remove the salt and pepper noise caused by rain and snow, and then Gaussian filtering was used to smooth the local brightness unevenness caused by backlighting, resulting in a clear image. Then, normalization and target recognition are performed: the image size is uniformly scaled to 640×480 pixels, the pixel values ​​are normalized to the [0,1] range, and the basic target detection algorithm is used to initially locate targets such as "vehicles" and "pedestrians".

[0035] After preprocessing, city scene labels are generated using automatic labeling tools or manually. Labeling information: Target type: Illegal street vending (blue truck); Target location: Bottom left corner of image (coordinates: (120,350)-(280,480)); Scene tag: Traffic violation; Affected area: Occupying non-motorized vehicle lane.

[0036] Video data (subway morning rush hour passenger flow monitoring video), such as: 30 minutes of morning rush hour monitoring video of a subway station (including empty shots with no passengers in some periods, shaky footage, dense crowds, and minor obstructions).

[0037] First, invalid data is removed: based on scene recognition algorithms (preset scene feature library such as "empty shot", "no target" etc.), 10-minute empty shot segment with no passenger flow at 5 am is removed. Secondly, frame extraction is performed: frames are extracted at a rate of 1 frame / second, and a total of 1200 frames are extracted from 30 minutes of effective video (800 frames are actually extracted after removing empty shots). Furthermore, image processing after frame extraction (such as preprocessing methods for image-type data): each extracted frame image is denoised (Gaussian filtering) and its size is standardized (uniformed to 800×600 pixels), and the personnel area is initially selected through target localization algorithm.

[0038] After preprocessing, city scene labels are generated using automatic labeling tools or manually. Labeling information: Scenario type: Traffic flow (subway morning rush hour); Passenger density: High (≥5 people / m²) 2 Congested areas: central platform and escalator entrances; Warning suggestion: open temporary diversion channels.

[0039] Voice data (hotline recordings), such as a resident hotline recording (60 seconds long, including background noise, such as office noise, colloquial expressions, and some pauses in sentences): "Hello... I'd like to report that there are many vendors occupying the road at the west gate of XX Park, and their loudspeakers are extremely loud. They are still making noise at 10 pm, and it's impossible to rest..."

[0040] First, a quality assessment is conducted using a speech quality assessment algorithm (calculating signal-to-noise ratio and distortion). If the signal-to-noise ratio of the recording is ≥25dB, it is considered valid; if the recording is silent or has more than 50% noise, it is rejected. Secondly, noise reduction is performed: background noise is filtered out using the Wiener filtering algorithm to enhance the clarity of human voices; Then, standardization and speech-to-text processing were performed: the sampling rate was uniformly set at 16kHz, and the speech was converted into text: "There are vendors occupying the road at the west gate of XX Park, and the loudspeaker volume is too loud. At 10 pm, there is still a disturbance to the residents and it affects their rest."

[0041] After preprocessing, city scene labels are generated using automatic labeling tools or manually. Labeling information: Problem type: Urban management (street vendors + noise pollution); Time: 10 PM; Location: West gate of XX Park; Urgency level: Low; Request: Eliminate street vendors and control noise.

[0042] This embodiment achieves comprehensive and compliant collection of four core urban data types—text, images, video, and voice—by connecting with the business systems of various urban departments, terminal monitoring equipment, and citizen feedback channels. Simultaneously, relying on standardized interface specifications, hierarchical permission control, and differentiated collection frequency settings, it ensures the efficiency and security of data access. Furthermore, targeting the characteristics of different data types, specific preprocessing algorithms such as keyword matching, hash deduplication, median filtering, Gaussian filtering, Wiener filtering, and frame extraction are employed to effectively remove invalid information, reduce data noise, and unify data formats, thereby improving the quality and usability of the raw data. Then, through automatic annotation tools combined with manual verification, standardized urban scene labels are added to the preprocessed data, forming a high-quality, structured, and scenario-based labeled dataset. This not only lays a solid data foundation for the subsequent construction of three types of specialized datasets and the training and fine-tuning of models but also significantly improves the adaptability of data to the needs of urban governance, providing accurate and reliable data support for the intelligent operation of urban infrastructure platforms.

[0043] Optionally, in one specific embodiment, the urban multimodal data includes urban data of multiple different data types; wherein, the data types include text type, image type, video type, and voice type; Based on annotated urban multimodal data, a question-answering sample set, a scene matching dataset, and a preference dataset were constructed, including: Based on the data type and label of each city in the annotated multimodal data of cities, all city data are classified to divide each city data into text-dominant data, multimodal correlated data and user feedback correlated data, and the same city data can be one or more of the following: text-dominant data, multimodal correlated data and user feedback correlated data. Based on a pre-set prompt word generation tool, question-answer pairs are constructed for each text-dominant data to build a question-answer sample set; Based on the data type and label corresponding to each multimodal association data, multimodal association data with the same scene label but belonging to different data types are paired to construct a scene pairing dataset. The scene label is obtained by feature extraction from the label of the multimodal association data. A preference dataset is constructed based on the data type and labeling of each user's feedback data.

[0044] Specifically, for labeled text, image, video, and audio multimodal data, combined with their original data type (e.g., text / image) and labeling tags (e.g., "manhole cover malfunction," "street vending," "user satisfaction"), multi-dimensional classification is performed to determine the attribution of each data item, such as: Text-driven data refers to labeled data with complete textual information that can independently support question-and-answer tasks. The core of this type of data is text-based data, but it also includes text with complete information after speech-to-text conversion. Examples include citizen complaint texts labeled "facility malfunction (manhole cover)", government policy texts labeled "traffic violation handling procedures", and hotline texts labeled "noise disturbance complaints" after speech-to-text conversion.

[0045] Multimodal correlated data refers to labeled data containing information about the same scene but belonging to different data types. Cross-type linkage is required to fully present the scene. For example: a surveillance image of an intersection labeled "illegal street vending" (image type) and the corresponding citizen complaint text (text type); a frame of a surveillance video labeled "metro morning rush hour congestion" (image type) and time-series passenger flow data (extended data type); a drone aerial image labeled "illegal construction investigation" (image type) and law enforcement record video clips (video type).

[0046] User feedback-related data: This refers to labeled data containing user feedback on handling results, service quality, etc., and its core consists of "original problem data + user feedback tags". For example: complaint work order text labeled "Manhole cover malfunction handling - user satisfaction", surveillance image labeled "Street vendor identification result - law enforcement personnel approval", and hotline voice-to-text text labeled "Consultation reply - citizen dissatisfaction".

[0047] Note: The same data can be categorized across different categories. For example, citizen complaint texts labeled "illegible street vending" (text-dominated data) can be combined with corresponding surveillance images to form multimodal associated data. If citizen feedback on the handling results is attached, it can also be classified as user feedback associated data.

[0048] Based on pre-set prompt word generation tools (such as prompt word engines built on existing models), question-and-answer pairs are constructed for each piece of text-based data. The specific process is as follows: The prompt word generation tool has built-in urban scenario question-and-answer templates. For example, "For questions marked with {label}, generate questions from a citizen's perspective, and generate answers based on labeled information / government rules," and "Based on {text content}, extract core questions and corresponding solutions." Inputting text-based data, the tool automatically generates question-and-answer pairs: If the text is "The manhole cover at the east gate of XX Road XX Community is damaged, discovered last week, posing a hazard to pedestrians" (labeled "Facility Failure (Manhole Cover)"), the generated question is "How should the damaged manhole cover at the east gate of XX Road XX Community be handled?", and the answer is "Contact the municipal maintenance department; a dispatch order will be sent within 30 minutes, and repairs will be completed within 24 hours to ensure pedestrian safety." If the text is "Traffic violation fine payment process: online through the government affairs APP, offline at the vehicle management office" (labeled "Traffic Violation Handling"), the generated question is "What are the payment methods for traffic violation fines?", and the answer is "Pay online through the government affairs APP, and offline at the vehicle management office." The generated question-and-answer pairs are manually verified to remove samples with ambiguous semantics or answers that do not match the labeled information, and finally integrated to form a question-and-answer sample set.

[0049] By employing multimodal data association algorithms (such as cosine similarity matching algorithms based on scene labels and graph neural network association algorithms), accurate pairing of different types of data within the same scene is achieved. Scene labels are extracted: for all city data, NLP feature extraction algorithms (such as TF-IDF and BERT vector encoding) are used to extract unified scene labels from the labeled tags. For example, the core scene label "street vending" is extracted from labels such as "street vending (blue truck)" and "citizen complaints about street vending"; the scene label "street vending during morning rush hour" is extracted from labels such as "metro morning rush hour congestion" and "high passenger density". Through multimodal data association algorithms, multimodal data with consistent scene labels but different data types are paired in pairs or multiple groups to form paired samples of "scene + multiple data types". For example, image-text pairing (surveillance images labeled "street vending" + corresponding video complaint text labeled "street vending") and video-text pairing (enforcement video clips labeled "illegal construction investigation" + corresponding aerial images + enforcement record text). Each paired sample is labeled with a unified scene label and associated description, and integrated to form a scene pairing dataset, which can meet the cross-type data processing needs of multimodal models.

[0050] Based on pre-defined priority strategies (such as response speed priority, handling effect priority, and user satisfaction priority), the user feedback data is processed, specifically including: Extracting core information: From each piece of user feedback data, the following are separated: "Original problem information (such as manhole cover malfunction complaint)," "Handling / identification result (such as dispatch repair)," "User feedback tags (such as satisfied / unsatisfied)," and "Priority labeling (such as high / medium / low)." Quantifying preferences according to priority strategies: Based on pre-defined strategies, user feedback is transformed into trainable preference labels. For example, if the pre-defined strategies are "Response time < 30 minutes → high priority" and "User satisfaction → high preference weight," then data labeled "Manhole cover malfunction - 30-minute dispatch - User satisfaction" is marked as "high preference," and data labeled "Consultation reply - 2-hour response - Citizen dissatisfaction" is marked as "low preference." Integrating to form a preference dataset: Each sample contains "Original multimodal data (text / image, etc.)," ​​"Handling / identification result," "Preference label (high / medium / low)," and "Priority description," adapting for subsequent model preference alignment fine-tuning.

[0051] This embodiment categorizes data into three types based on data type and label, ensuring the independence of single-type data and the correlation between multiple types of data while adapting to different dataset construction needs. Secondly, it generates accurate question-answer pairs using prompt word tools, combines algorithms to achieve pairing of multiple types of data within the same scenario, and quantifies user preferences based on priority strategies to construct three high-quality datasets. This not only achieves accurate adaptation of data to urban scenarios and model tasks but also provides targeted data support for subsequent model training and fine-tuning, improving the model's ability to adapt to urban business needs and practical application preferences.

[0052] Optionally, in a specific embodiment, the urban scene task support model and the large language model, which have been initially trained, are fine-tuned based on the preference dataset, including: Based on the preference dataset, the urban scene task support model and the language large model after initial training were subjected to structured pruning. Furthermore, based on the preference dataset, the structured pruning urban scene task support model and the language big model were fine-tuned respectively.

[0053] Furthermore, the structured pruning process includes: Identify the fully connected layers in the urban scene task support model or language large model that require pruning; Based on the pre-set low-rank dimension, the first low-rank matrix and the second low-rank matrix are inserted into the fully connected layer that needs to be pruned in order to update the urban scene task support model or language large model. The dimension of the first low-rank matrix is ​​the input feature dimension × low-rank dimension, and the dimension of the second low-rank matrix is ​​the low-rank dimension × output feature dimension. The preference dataset is input into the updated urban scene task support model or language large model for training. The first update gradient corresponding to the first low-rank matrix and the first update gradient corresponding to the second low-rank matrix are obtained through the backpropagation algorithm. Based on the first update gradient corresponding to the first low-rank matrix and the first update gradient corresponding to the second low-rank matrix, some model parameters in the urban scene task support model or language large model are removed.

[0054] Furthermore, update the urban scene task support model or language large model, including: For the fully connected layer in the urban scene task support model or language large model, the input-output relationship of the fully connected layer is reconstructed based on the first low-rank matrix, the second low-rank matrix, and a pre-defined formula (Formula 1). Formula 1 is: Y = X·w0 + A·B; Where Y is the output of the fully connected layer, X is the input of the fully connected layer, w0 is the pre-set initial weight, A is the first low-rank matrix, and B is the second low-rank matrix.

[0055] Furthermore, when eliminating some model parameters, the model parameters in the attention layer and FFN layer of some models in the urban scene task support model or the language big model are grouped, and the importance of all model parameters in each group is weighted and summed to obtain the corresponding group importance, so that model parameters with group importance lower than the pre-set importance threshold are eliminated.

[0056] Specifically, we first identify the fully connected layers with high parameter count and computational complexity in urban scene task support models (such as multimodal detection and temporal prediction models) and large language models (such as dialogue and solution generation models), and at the same time, we identify the attention layer and FFN layer that need to be optimized (in preparation for subsequent parameter grouping).

[0057] Preset the low-rank dimension (e.g., set the low-rank dimension to 256 or 512 according to the input and output feature dimensions of the model), and construct the first low-rank matrix A (dimension: input feature dimension × low-rank dimension) and the second low-rank matrix B (dimension: low-rank dimension × output feature dimension). According to Formula 1: Y=X·w0+A·B, the first low-rank matrix and the second low-rank matrix are inserted into the target fully connected layer to reconstruct the input-output relationship of the fully connected layer. Here, X is the input of the fully connected layer, Y is the output of the fully connected layer, w0 is the pre-set initial weight of the fully connected layer, and A·B is the product term of the first low-rank matrix and the second low-rank matrix.

[0058] The preference dataset (containing multimodal samples with city governance preference labels) is input into the updated model for training. Using the backpropagation algorithm, the first update gradients of the first low-rank matrix A and the second low-rank matrix B are calculated to reflect the parameter update trends in the preference task. The first and second update gradients are obtained by calculating the first loss value during model training, specifically by calculating the partial derivatives of the first loss value with respect to the first and second low-rank matrices. During the calculation, the initial weights are fixed to be non-differentiable to ensure that only the low-rank matrices capture the correlation of task weights.

[0059] The importance of model parameters is evaluated based on the first / second update gradient and weight values. The importance of any model parameter is the absolute value of the product of the model parameter and the first / second update gradient corresponding to the model parameter.

[0060] The contribution of each parameter in the fully connected layer (including the initial weights and elements in the low-rank matrix) to the model is calculated. Parameters below the threshold are removed. The weights of the pruned model are the sum of the product of the initial weights and the retained low-rank matrix.

[0061] Based on the structured and pruned model, the model parameters are further optimized using a preference dataset so that the model output aligns with the priority strategies of urban governance (such as prioritizing response speed and user satisfaction). For fine-tuning data preparation, the preference dataset is divided according to scenario type (such as manhole cover malfunction, street vendors), and each sample is labeled with a preference label (such as "high preference: dispatch within 30 minutes", "low preference: response within 2 hours"), which serves as a supervisory signal for fine-tuning.

[0062] The urban scene task support model / language large model that has undergone structured pruning is used as the initial policy network. The basic parameters retained after pruning are fixed, and only the update permissions of the first low-rank matrix A and the second low-rank matrix B are opened.

[0063] The preference dataset samples are input into the initial policy network to generate candidate outputs (such as fault handling solutions and complaint response texts); the reward value of the output is calculated by combining the preference labels (if it matches the high preference label, the reward value is high), and the reward signal is transmitted to the first low-rank matrix / second low-rank matrix through backpropagation to adjust the matrix parameters; Repeat the above input-output-parameter update process. After each iteration, use the validation set to test the matching degree between the model output and the preference label until the model output stably matches the city governance preferences (e.g., matching degree ≥ 85%), and complete the final model fine-tuning.

[0064] This embodiment achieves model lightweighting through structured pruning (reducing the number of parameters and inference time to adapt to the deployment requirements of the city platform), and also aligns the model's capabilities with the actual business preferences of city governance through preference dataset-driven fine-tuning, thus balancing the model's efficiency and practicality.

[0065] Furthermore, in a specific embodiment, the structured pruned urban scene task support model and the large language model are fine-tuned separately based on the preference dataset, including: The structured and pruned urban scene task support model or language large model is used as the initial policy network, and the reward function corresponding to the city data is constructed based on the label corresponding to each city data in the preference dataset. Data from any city in the preference dataset is used as input to the initial policy network to generate multiple candidate outputs; Based on the similarity between the reward function corresponding to the city data input to the initial policy network and each candidate output, and using the pre-set Formula 2, the reward parameter corresponding to each candidate output is obtained; Formula 2 is: ; Where R is the reward parameter and P is the similarity between the candidate output and the reward function; Based on a pre-set preference optimization algorithm and reward parameters, the first and second low-rank matrices in the initial policy network are optimized for model fine-tuning.

[0066] Based on this, and using a pre-set preference optimization algorithm and reward parameters, the first and second low-rank matrices in the initial policy network are optimized for model fine-tuning, including: Based on the pre-set preference optimization algorithm and reward parameters, the corresponding target loss function is obtained, and the second update gradient of the target loss function with respect to the first low-rank matrix and the second low-rank matrix is ​​obtained through the backpropagation algorithm. The first low-rank matrix and the second gradient matrix are updated based on the second update gradient of the first low-rank matrix and the second low-rank matrix, and the AdamW optimization algorithm is used to fine-tune the model.

[0067] Specifically, the urban scene task support model (such as multimodal scene detection and passenger flow prediction model) and the language big model (such as government affairs question answering and disposal plan generation model) that have completed structured pruning are used as two independent initial policy networks. The basic parameters retained after pruning (such as attention layer, FFN layer remaining parameters, and initial weight w0 of fully connected layer) are fixed, and only the update permissions of the first low-rank matrix A and the second low-rank matrix B are opened to ensure that the fine-tuning process does not destroy the lightweight structure of the model.

[0068] For each piece of city data in the preference dataset (such as complaint text labeled "manhole cover malfunction - dispatch within 30 minutes - user satisfaction" and surveillance images labeled "street vending - rapid response - law enforcement approval"), the core preference information in the labels is extracted (such as "response time < 30 minutes", "priority to handling effect", "high user satisfaction"). These preference rules are then transformed into digital reward functions. For example, for manhole cover malfunction data, the reward function can be defined as f(x) = 1 (if the output includes "dispatch within 30 minutes + 24-hour repair"), f(x) = 0 (if the output does not contain this content); for street vending data, the reward function can be defined as f(x) = 1 (if the output includes "on-site enforcement + cleanup within 1 hour"), f(x) = 0 (otherwise), thus quantifying whether the candidate outputs conform to the preferences.

[0069] Input any city data from the preference dataset (such as the text "complaint about damaged manhole cover on XX Road" or the image "street vendors occupying the road at XX intersection") into the corresponding initial policy network, and let the model generate multiple candidate outputs. For example, the urban scene task support model generates candidate results such as "identify the location of the vehicle occupying the road + suggest cleaning within 1 hour" and "identify only the location of the vehicle + suggest cleaning within 3 hours" for the image of street vendors occupying the road; the language large model generates candidate responses such as "contact the municipal government for repair within 30 minutes" and "repair order will be dispatched within 1 hour after reporting" for the text of manhole cover malfunction.

[0070] A similarity calculation algorithm (such as cosine similarity or Jaccard similarity) is used to compare the matching degree between each candidate output and the corresponding reward function to obtain a similarity P (with a value ranging from 0 to 1). For example, the similarity P between the candidate output "repair dispatch within 30 minutes" and the reward function "response time < 30 minutes" is 1; the similarity P between the candidate output "repair dispatch within 1 hour" and the reward function is 0.2.

[0071] Substituting the similarity P into the above formula R = logP / (1-P), which is expressed in a conventional logarithmic form, R reaches its maximum value if P = 1 and its minimum value if P = 0, yielding the reward parameter R for each candidate output. The magnitude of the reward parameter R directly reflects the degree to which the candidate output conforms to the preference; the larger the R value, the more closely the candidate output fits the preference rules of urban governance.

[0072] A preset preference optimization algorithm is selected, and a target loss function is constructed by combining the reward parameter R mentioned above. That is, the loss function L, constructed by combining the reward parameter R and the advantage function A(s,a), serves to constrain the magnitude of model parameter updates, ensuring stable training while gradually aligning the model output with urban governance preferences. Specifically: ; Where θ represents the model parameters to be updated, specifically the first and second low-rank matrices of the fully connected layer after structured pruning; θ,old represents the old parameters from the previous iteration, used to limit the parameter update magnitude; s represents samples in the input preference dataset; a represents the candidate outputs generated by the model for input s; and π represents the parameter values. θ (a|s) represents the probability of generating candidate output a when the model is input s, given the current parameters; π θ,old (a|s) is the probability of generating candidate output a when the model inputs s under the parameters of the previous round. To measure the change in the probability of the model generating the same output before and after parameter updates, A(s,a) is the advantage function, which quantifies the preference advantage of candidate output a relative to the average level. It is calculated using the reward parameter R, with the formula A(s,a) = R(s,a) - R. 平均 (s), where R(s,a) is the reward parameter corresponding to the output a generated from the input s, calculated by formula 2, R 平均 A(s) represents the mean of the reward parameters for all candidate outputs generated by the model when input s. If A(s,a) > 0, it means that the output a is better than the average level, and the model needs to increase the probability of generating this output. Conversely, the probability needs to be reduced. This is the clipping factor, typically ranging from 0.1 to 0.3, which limits the probability ratio to a certain value. This avoids excessively large parameter update fluctuations. `clip` is the clipping function.

[0073] The low-rank matrices A and B after structured pruning are used as initial parameters θ. old Set the clipping factor =0.2, learning rate and other hyperparameters. Input the preference dataset sample s into the model to generate candidate output a, calculate the similarity P between the output and the reward function, substitute it into the model to obtain R(s,a), and then calculate the average reward of all candidate outputs under the same output to obtain A(s,a).

[0074] Substitute the values ​​into the L(θ) formula above to calculate the loss, and then use the backpropagation algorithm to calculate the gradient of the loss function with respect to the low-rank matrices A and B.

[0075] Update the parameter values ​​of A and B along the gradient descent direction to complete one iteration; every 10-20 iterations, change θ. old Update to θ and repeat the above steps until the model output matches the preference label to the threshold.

[0076] This embodiment uses a lightweight model with structured pruning as the initial policy network, fixing the basic parameters and only granting access to low-rank matrices for updates. This retains the advantages of lightweight models, reduces inference time and deployment costs, and avoids damaging the core structure of the model during fine-tuning. The pruning mechanism avoids training oscillations, and the reward parameters incorporate urban governance preferences (such as "30-minute dispatch" and "rapid response") into the optimization objective, ensuring that the pruned lightweight model outputs results that better meet actual business needs while maintaining inference efficiency.

[0077] In addition, this application provides an urban infrastructure platform with the following structure: Figure 2 As shown, it includes a data processing layer, a model management layer, a model training layer, a model evaluation layer, a model deployment layer, a service layer, and a knowledge base. Its construction method is as follows: Figure 3 As shown, it includes: Data Access and Processing: The implementation plan involves initial screening and extraction of various raw data (including images, videos, text, and other formats) from the collected data. Specifically, for video data, frame extraction is performed to convert it into a static image sequence. Simultaneously, valid data samples containing specific scenes (such as violations in subway stations) need to be accurately identified and filtered from massive amounts of video material. Subsequently, detailed annotation is performed on this preprocessed data, strictly adhering to the training data format specifications required for large-scale model fine-tuning. Annotation can be done in two ways: automated annotation using algorithms or manual annotation by professionals. To ensure data quality, all annotated data undergoes rigorous review and verification by a human team. Finally, through these systematic processing steps, a high-quality data sample set meeting the model training requirements is obtained.

[0078] Dataset Construction: The implementation plan includes an online annotation platform supporting manual annotation of text, images, and audio data, enabling model- or template-based annotation, and providing templates for question-and-answer and text / image annotations. For constructing various datasets, a prompt word generation tool is developed, combining data generation models to construct datasets, collecting real and simulated data to build dialogue scenario datasets, integrating text and image resources to build multimodal scenario datasets, and collecting preference information through user feedback to build preference datasets for alignment fine-tuning. During dataset segmentation, the tool supports hierarchical random segmentation according to specified ratios or specific features and generates metadata. For multi-dataset sampling ratios, a function allows users to select multiple datasets and set proportions for random sampling and merging. Regarding sub-dataset construction, the tool allows users to filter the original dataset according to specific conditions to generate sub-datasets, and these sub-datasets support further annotation and segmentation operations, comprehensively meeting the data needs for model training and evaluation.

[0079] Model Management: For model repository construction, a fully functional storage system is built, pre-integrating diverse models with detailed information; flexible model access methods are provided, supporting both local and remote connections; an easy-to-use interface is developed to enable basic model operations and facilitate data flow and collaboration with core processes. Regarding model version management, a version tracking mechanism is constructed to record change information, supporting version comparison and rollback; a version storage and maintenance system is established to save models and parameters, with file format conversion capabilities, ensuring orderly, efficient, and flexible model management.

[0080] Model Development: In terms of training framework and methods, it integrates mainstream distributed training frameworks such as DeepSpeed ​​and Megatron, supporting distributed training modes such as single-machine multi-GPU and multi-machine multi-GPU, leveraging the advantages of hardware cluster computing. Regarding training optimization techniques, it implements diverse parallel computing methods to improve computational parallelism and memory utilization, introducing performance optimization techniques to accelerate training and save resources. For model training stability and continuity, it provides a breakpoint handling mechanism, supporting breakpoint saving, training continuation, and retraining. In terms of development convenience, it provides rich model training templates and descriptions, supports flexible configuration of resource parameters, and provides a customizable development environment via image. Furthermore, it builds a training task management system to record, save, and manage training history information, facilitating traceability and analysis, and supporting continuous optimization of model development.

[0081] Model Pre-training and Fine-tuning: Model Pre-training Stage and Configuration: A flexible and scalable training architecture is built, supporting pre-training from scratch to build new models and continuing pre-training to optimize existing models; a vocabulary expansion tool is developed, which can expand the Chinese vocabulary on demand to process Chinese data and expand the vocabulary of special symbols to meet specific domain needs. Model Fine-tuning Stage and Configuration: Stable and efficient supervised model fine-tuning functions are provided to help models quickly adapt to new tasks; algorithms such as reinforcement learning based on human feedback are introduced to achieve model alignment fine-tuning so that the output meets requirements; full parameter fine-tuning and efficient parameter fine-tuning algorithms such as LoRA and P-tuning are supported to meet the needs of different scenarios. Algorithm Parameter Tuning: An intuitive and easy-to-use parameter configuration interface is created, supporting flexible modification of common parameters such as the learning rate. Specific parameter setting entries are provided for different fine-tuning algorithms, such as low-rank matrices, for users to fine-tune and improve the effectiveness and efficiency of large model pre-training and fine-tuning.

[0082] Model Evaluation: In terms of evaluation dimensions, the system provides basic performance evaluation functions covering common metrics such as accuracy and F1 score to accurately measure model performance. It also supports user-defined evaluation metrics to meet diverse evaluation requirements. Regarding evaluation methods, the system supports evaluation based on different types of datasets, offers multiple evaluation methods for users to choose from, and provides rich templates to ensure consistent and stable evaluation results. For result presentation, the system automatically generates detailed reports or logs, recording the evaluation process and key data. Simultaneously, it incorporates visualization technology to display results in charts and other formats, facilitating user viewing of results, model comparison, and providing support for optimization and decision-making.

[0083] Model Deployment and Inference: In the model deployment phase, a flexible resource configuration platform is built, allowing users to specify key parameters such as resource type, quantity, and runtime dependencies as needed, ensuring the model runs in the appropriate environment. It also provides deployment methods such as single-machine multi-GPU and multi-machine multi-GPU to meet the needs of different business scenarios. For model inference, a comprehensive inference service management system is constructed, supporting real-time querying and display of service information such as service version and instance count, allowing users to easily monitor service status. Furthermore, an inference service orchestration function is introduced, supporting the orderly orchestration of multiple models, enabling deep collaboration between inference services and other modules, improving business processing efficiency and intelligence, and providing users with a smooth and reliable experience.

[0084] Knowledge Base Construction: First, the modular data access layer supports real-time extraction from data sources such as relational databases (MySQL / Oracle), non-relational databases (Elasticsearch / Neo4j / vector databases), web pages, and APIs, and is compatible with parsing and structured storage of multimedia formats such as text (TXT / PDF / Excel), images (PNG / JPG), and audio / video (MP4 / WAV). Next, in the knowledge processing layer, NLP technology is used for domain text entity recognition, relation extraction, and attribute normalization. A rule engine and machine learning model are combined to build and dynamically update the knowledge graph. Embedding model services (such as BERT and Sentence-BERT) are integrated to encode text slices into high-dimensional vectors, enhancing semantic retrieval. Finally, the system provides a visual knowledge management interface, supporting field-level metadata customization, sample data batch processing, and graph topology analysis. A built-in vector index engine enables millisecond-level similarity retrieval, forming a scalable domain knowledge hub that provides structured knowledge and semantic information enhancement capabilities for scenarios such as intelligent question answering and decision support.

[0085] API Service Setup: At the service interface level, it provides a rich set of standardized interface types, including SDK, HTTP API, and gRPC API. Dedicated APIs are provided for different inference service scenarios such as dialogue, vector, and image generation to meet business needs. Regarding service parameter configuration, it supports detailed settings for service, model, and application parameters to adapt to different scenarios. Interaction methods include coded, low-code, and no-code modes, lowering the barrier to entry. Rigorous testing is conducted before deployment, with models released in a simulated environment to verify inference results and ensure service stability. For resource management, it provides unified and efficient management of AI resources such as datasets and model repositories. Furthermore, to improve service efficiency and reliability, it supports collaborative orchestration of different models, enabling flexible calling of models of varying sizes and allowing for the orchestration of various objects such as models and external components, building a collaborative and orderly service ecosystem.

[0086] First, to address the issue that traditional platforms typically process text, images, and sensor data (such as camera and GPS data) independently and lack cross-modal correlation capabilities, this platform supports the linkage of multi-source, multi-modal data to construct training samples, enabling models to adapt to different data for analysis. Second, traditional platforms require the deployment of different visual and language models to meet varying needs, lacking collaborative capabilities. This platform can develop large-scale models based on diverse data, and the trained models can directly meet the needs of different scenarios. Finally, traditional knowledge bases rely on manual maintenance and cannot synchronize with real-time urban changes (such as newly built roads or policy adjustments). The platform in this embodiment can access image, text, and voice data in real time and perform knowledge extraction and cleaning based on knowledge graphs, achieving timely and accurate updates to the knowledge base.

[0087] In addition, this application provides a system for building an urban infrastructure platform, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the above-described method for building an urban infrastructure platform.

[0088] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0089] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0090] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0091] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0092] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for constructing an urban infrastructure platform, characterized in that, include: Acquire urban multimodal data and perform annotation processing on the urban multimodal data; Based on the annotated urban multimodal data, a question-answering sample set, a scene matching dataset, and a preference dataset are constructed. The question-answering sample set is a dataset constructed using a prompt word generation algorithm, the scene matching dataset is a dataset constructed using a multimodal data association algorithm, and the preference dataset is a dataset constructed using a pre-set priority strategy. The urban scene task support model pre-deployed in the model repository is initially trained based on the scene matching dataset, and the language large model pre-deployed in the model repository is initially trained based on the question answering sample set. The urban scene task support model and the language large model are fine-tuned based on the preference dataset to obtain the trained urban scene task support model and language large model. Based on the annotated urban multimodal data, a structured database and an unstructured database are constructed, and an NLP algorithm is used to perform entity recognition, relationship processing and attribute normalization on the urban multimodal data in order to construct a knowledge graph. Based on the pre-set API interface, API services are built for the trained urban scene task support model and language big model, as well as structured databases, unstructured databases and knowledge graphs, to obtain a completed urban infrastructure platform. Based on the preference dataset, the urban scene task support model and the large language model were fine-tuned after initial training, including: Based on the preference dataset, the urban scene task support model and the language large model after initial training were subjected to structured pruning. Furthermore, based on the preference dataset, the structured pruning urban scene task support model and the language big model were fine-tuned respectively.

2. The method for constructing an urban infrastructure platform according to claim 1, characterized in that, Acquire urban multimodal data and perform annotation processing on the urban multimodal data, including: Acquire urban multimodal data and preprocess the urban multimodal data based on the data types, including text, image, video, and audio. The preprocessed urban multimodal data is then labeled. The preprocessing includes: The invalid information of the text-type urban multimodal data is removed based on the keyword matching algorithm, and the text-type urban multimodal data is deduplicated based on the pre-set hash algorithm. Then, the deduplicated text-type urban multimodal data is cleaned for noise, text standardization and format conversion. Invalid information is removed from the image-type urban multimodal data based on the image quality assessment algorithm. After denoising the image-type urban multimodal data using median filtering and Gaussian filtering algorithms, the image-type urban multimodal data is normalized and target recognition is performed. Invalid data in urban multimodal data of video type is removed based on scene recognition algorithm, and frame extraction is performed on urban multimodal data of video type to perform noise reduction, size standardization and target localization on the extracted image; The speech quality assessment algorithm is used to evaluate the quality of urban multimodal data of speech type. After removing noise from the speech type urban multimodal data using the Wiener filtering algorithm, the speech type urban multimodal data is standardized and processed for speech-to-text conversion.

3. The method for constructing an urban infrastructure platform according to claim 1, characterized in that, The urban multimodal data includes urban data of multiple different data types; among them, the data types include text, image, video and audio. Based on annotated urban multimodal data, a question-answering sample set, a scene matching dataset, and a preference dataset were constructed, including: Based on the data type and label of each city in the annotated multimodal data of cities, all city data are classified to divide each city data into text-dominant data, multimodal correlated data and user feedback correlated data, and the same city data can be one or more of the following: text-dominant data, multimodal correlated data and user feedback correlated data. Based on a pre-set prompt word generation tool, question-answer pairs are constructed for each text-dominant data to build a question-answer sample set; Based on the data type and label corresponding to each multimodal association data, multimodal association data with the same scene label but belonging to different data types are paired to construct a scene pairing dataset. The scene label is obtained by feature extraction from the label of the multimodal association data. A preference dataset is constructed based on the data type and labeling of each user's feedback data.

4. The method for constructing an urban infrastructure platform according to claim 1, characterized in that, The process of structured pruning includes: Identify the fully connected layers in the urban scene task support model or language large model that require pruning; Based on the pre-set low-rank dimension, the first low-rank matrix and the second low-rank matrix are inserted into the fully connected layer that needs to be pruned in order to update the urban scene task support model or language large model. The dimension of the first low-rank matrix is ​​the input feature dimension × low-rank dimension, and the dimension of the second low-rank matrix is ​​the low-rank dimension × output feature dimension. The preference dataset is input into the updated urban scene task support model or language large model for training. The first update gradient corresponding to the first low-rank matrix and the first update gradient corresponding to the second low-rank matrix are obtained through the backpropagation algorithm. Based on the first update gradient corresponding to the first low-rank matrix and the first update gradient corresponding to the second low-rank matrix, some model parameters in the urban scene task support model or language large model are removed.

5. The method for constructing an urban infrastructure platform according to claim 4, characterized in that, Update the urban scene task support model or language large model, including: For the fully connected layer in the urban scene task support model or language large model, the input-output relationship of the fully connected layer is reconstructed based on the first low-rank matrix, the second low-rank matrix, and a pre-set formula 1; the formula 1 is: Y = X·w0 + A·B; Where Y is the output of the fully connected layer, X is the input of the fully connected layer, w0 is the pre-set initial weight, A is the first low-rank matrix, and B is the second low-rank matrix.

6. The method for constructing an urban infrastructure platform according to claim 4, characterized in that, When eliminating some model parameters, the model parameters in the attention layer and FFN layer of some models in the urban scene task support model or language large model are grouped, and the importance of all model parameters in each group is weighted and summed to obtain the corresponding group importance, so that model parameters with group importance lower than the preset importance threshold are eliminated.

7. The method for constructing an urban infrastructure platform according to claim 5, characterized in that, Based on the preference dataset, the structured pruned urban scene task support model and the large language model were fine-tuned separately, including: The structured and pruned urban scene task support model or language large model is used as the initial policy network, and the reward function corresponding to the city data is constructed based on the label corresponding to each city data in the preference dataset. Data from any city in the preference dataset is used as input to the initial policy network to generate multiple candidate outputs; Based on the similarity between the reward function corresponding to the city data input to the initial policy network and each candidate output, and using a pre-set formula two, the reward parameter corresponding to each candidate output is obtained; the formula two is: ； Where R is the reward parameter and P is the similarity between the candidate output and the reward function; Based on a pre-set preference optimization algorithm and reward parameters, the first and second low-rank matrices in the initial policy network are optimized for model fine-tuning.

8. The method for constructing an urban infrastructure platform according to claim 7, characterized in that, Based on a pre-set preference optimization algorithm and reward parameters, the first and second low-rank matrices in the initial policy network are optimized for model fine-tuning, including: Based on the pre-set preference optimization algorithm and reward parameters, the corresponding target loss function is obtained, and the second update gradient of the target loss function with respect to the first low-rank matrix and the second low-rank matrix is ​​obtained through the backpropagation algorithm. The first low-rank matrix and the second gradient matrix are updated based on the second update gradient of the first low-rank matrix and the second low-rank matrix, and the AdamW optimization algorithm is used to fine-tune the model.

9. A system for building an urban infrastructure platform, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method for building an urban infrastructure platform as described in any one of claims 1 to 8.

Images