Method for quickly reporting city management problem based on multi-modal large model

Abstract

According to the city management problem rapid reporting method based on the multi-modal large model, advanced technologies such as image recognition, natural language processing and deep learning are integrated, related event information can be efficiently extracted and recognized from photos, the accuracy of problem recognition is improved, meanwhile, the efficiency is improved, and the method is suitable for popularization and application. The problem information in the event picture is accurately extracted, the to-be-confirmed information can be automatically backfilled by the large model, and the reporting process can be completed by performing simple check and confirmation after the user receives the automatically generated form, so that the links of manual intervention and data entry are reduced, the error probability is reduced, and the event reporting speed and efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent urban management, and more specifically, to a method for quickly reporting urban management problems based on a multimodal large model. Background Art

[0002] In modern urban management, the rapid identification and handling of problems is the key to ensuring the efficiency of urban operations and the quality of life of citizens. With the continuous expansion of urban scale and the increase of population, the challenges faced by urban management are also increasing. For example, problems such as road damage, facility failure, and environmental pollution not only affect the life experience of citizens, but may also cause safety hazards.

[0003] Traditional methods of reporting urban management issues usually have some potential drawbacks:

[0004] 1. Manual filling error

[0005] In the traditional process of reporting urban management issues, various errors often occur when manually filling in information, such as typos, missing information, or inaccurate information. These errors may result in the inability to accurately describe or record the problem, thus affecting the accuracy and efficiency of subsequent processing. Especially under high pressure, staff may make mistakes due to fatigue or negligence, further exacerbating the unreliability of data.

[0006] 2. Incomplete information

[0007] Due to the possibility of omissions or lack of detailed descriptions during manual filling, the submitted problem information is often incomplete. This will prevent the handlers from fully understanding the specific circumstances of the problem, thus affecting the subsequent investigation and resolution process. The lack of complete information will lead to inaccurate solutions and even require multiple rounds of confirmation.

[0008] 3. Poor user experience

[0009] The cumbersome filling process often leads to a poor user experience and affects citizens' enthusiasm for reporting problems. Complex forms and heavy information input requirements may frustrate users, thereby reducing their willingness to report problems. This negative experience may reduce the number and quality of problem reports.

[0010] The prior art (publication number: CN110020755B publication date: 2021.01.29) discloses an urban management system based on human-machine collaboration, which discloses: analyzing various types of information in urban management historical files, wherein the various types of information in urban management historical files include the type of file, the longitude and latitude corresponding to the file, the time when the file occurred, the description of the file, the picture information of the file, the environment corresponding to the file, and the social factors corresponding to the file. However, the problem information after analysis is directly reported, and it is impossible to avoid the situation where the information in the analysis process is incorrect, and the accuracy of problem identification may be low.

[0011] The prior art (publication number: CN106251271A publication date: 2016.12.21) discloses a city intelligent management platform, which specifically discloses the level of events and divides them according to importance and urgency. After the events are uploaded to the control center, the administrator needs to divide and process them, which still requires a lot of manual operations and has a low degree of automation.

[0012] The prior art (publication number: CN111507413B publication date: 2023.06.16) discloses a method for urban management case image recognition based on dictionary learning, which discloses that after the test samples are classified into known case categories, the urban management cases are classified, and after the case is identified, the results of the algorithm recognition will be combined with the video location, case description, time of the incident and other integrated information and published on the case processing platform after review. The system may directly report the case information after analyzing and integrating it, which may result in incorrect information and affect the progress of subsequent problem handling. The accuracy of the identification problem is unknown.

[0013] Therefore, efficiently and accurately collecting and processing this problem information has become an important task to improve the level of urban management. It is very necessary to provide a rapid reporting method for urban management problems based on a multimodal large model that can improve the accuracy of problem identification and reporting efficiency and improve the overall efficiency of urban management. Summary of the invention

[0014] In view of this, the present invention provides a method for quickly reporting urban management issues based on a multimodal large model, comprising the steps of:

[0015] Users upload event pictures to the cloud server through the client;

[0016] The cloud server stores event images;

[0017] The recognition device recognizes that the event image is an urban management issue and generates an event description item, which includes: event title, event level, event category and event subcategory;

[0018] The recognition device sends a request to the browser to call the big model, which is a model trained by combining text and image information. The big model includes: a multimodal language model, a machine learning model, and a deep learning model;

[0019] The recognition device transmits the event picture and the event description item to the large model;

[0020] The large model receives and parses the event image and event description items to obtain the information to be confirmed. The information to be confirmed includes the event description information and event location information, including:

[0021] The machine learning model and deep learning model preprocess the event images, including: resizing, color space conversion, and denoising;

[0022] Machine learning models and deep learning models extract key elements of event images, including color and content;

[0023] The machine learning model and the deep learning model compare the key elements of the event image with the key elements of the preset images in the preset template library, and select the preset image whose similarity with the key elements of the event image is greater than a preset value;

[0024] The machine learning model and the deep learning model read the preset event information of the corresponding preset image, wherein the preset event information includes: preset event level, preset event category and preset event subcategory; the preset event level includes: daily events, general events and major events; the preset event categories include: urban management, production safety and environmental sanitation; the preset event subcategories include public health, urban environment, publicity and advertising and construction management;

[0025] The machine learning model and the deep learning model traverse the event description items, write the preset event information into the corresponding event description items, and obtain the event description information;

[0026] The machine learning model and deep learning model read the event image and obtain the event location information;

[0027] The multimodal language model organizes the formats of event description information and event location information to obtain information to be confirmed;

[0028] The multimodal language model sends the information to be confirmed to the client;

[0029] The user checks the information to be confirmed and sends it to the city management problem platform;

[0030] The user checks the information to be confirmed. If it is incorrect, repeat the steps of uploading the event image to the cloud server through the client and the subsequent steps;

[0031] The user checks the information to be confirmed and cancels the report;

[0032] If the recognition device identifies the event image as a non-city management issue, a prompt is sent to the client.

[0033] Optionally, the event location information includes event description, event address, longitude and latitude.

[0034] Optionally, the client includes a preset form, and the multimodal language model sends the information to be confirmed to the client, including: the multimodal language model establishes a connection with the client; the multimodal language model fills the information to be confirmed into the preset form.

[0035] In a second aspect, the present invention provides a device for quickly reporting urban management issues based on a multimodal big model, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the device for quickly reporting urban management issues based on a multimodal big model executes a method for quickly reporting urban management issues based on a multimodal big model as described in any one of the first aspects.

[0036] In a third aspect, the present invention provides an electronic device comprising: a processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium and is characterized in that when the computer program is executed by the processor, it implements the method for quickly reporting urban management problems based on a multimodal large model of any one of claims 1 to 3.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for rapidly reporting urban management issues based on a multimodal large model according to any one of the first aspects.

[0038] Compared with the prior art, the method for quickly reporting urban management issues based on a multimodal large model provided by the present invention achieves at least the following beneficial effects:

[0039] The invention provides a method for quickly reporting urban management problems based on a multimodal big model, comprising the following steps: a user uploads an event picture to a cloud server through a client; the convenience of problem reporting is improved; the cloud server stores the event picture and generates a storage address; the event picture is identified as an urban management problem and an event description item is generated; the big model is called and the multimodal big model is used to further accurately analyze the urban management problem and improve the accuracy of problem identification; the event picture and the event description item are transmitted to the big model, the big model receives and parses the event picture and the event description item to obtain information to be confirmed; the big model reports the information to be confirmed to the client, the user checks the information to be confirmed, and if it is correctly submitted with the uploaded event picture information, the method integrates advanced technologies such as image recognition, natural language processing and deep learning, can efficiently extract and identify relevant event information from photos, improve the accuracy of problem identification and speed up efficiency, the problem information in the event picture is accurately extracted, the big model automatically fills in the information to be confirmed, and after receiving the automatically generated form, the user performs a simple check and confirmation to complete the reporting process, reduces the links of manual intervention and data entry, reduces the probability of errors, and effectively improves the speed and efficiency of event reporting.

[0040] Of course, any product implementing the present invention does not necessarily need to achieve all of the technical effects described above at the same time.

[0041] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0043] Figure 1 It is a flow chart of a method for quickly reporting urban management issues based on a multi-modal large model provided by the present invention;

[0044] Figure 2 It is a schematic diagram of the event picture;

[0045] Figure 3 This is another flow chart of a method for quickly reporting urban management issues based on a multi-modal large model provided by the present invention;

[0046] Figure 4 It is a schematic diagram of the preset form;

[0047] Figure 5 This is another flow chart of a method for quickly reporting urban management issues based on a multi-modal large model provided by the present invention;

[0048] Figure 6It is a schematic diagram of a device for quickly reporting urban management issues based on a multimodal large model provided by the present invention. DETAILED DESCRIPTION

[0049] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0050] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0051] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0052] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0053] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0054] Figure 6 A schematic diagram of the composition of a device for quickly reporting urban management issues based on a multi-modal large model provided by an embodiment of the present invention. Figure 6 As shown, the device for quickly reporting urban management issues based on a multimodal large model may include: at least one processor 11, a memory 12, a communication interface 13 and a communication bus 14.

[0055] Among them, the processor 11 is the control center of the urban management problem rapid reporting device based on the multimodal large model, which can be a CPU, a microprocessing unit, or one or more integrated circuits for controlling the execution of the program of the embodiment of the present invention.

[0056] As an embodiment, the processor 11 may include one or more CPUs, for example Figure 6 Furthermore, as an embodiment, the urban management problem rapid reporting device based on the multimodal large model may include multiple processors, such as Figure 6 The processor 11 and the processor 15 shown in FIG. Each of these processors can be a single-core processor (Single-CPU) or a multi-core processor (Multi-CPU).

[0057] The memory 12 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 12 may exist independently and be connected to the processor 11 via the communication bus 14. The memory 12 may also be integrated with the processor 11.

[0058] In a specific implementation, the memory 12 is used to store the data in the present invention and execute the software program of the present invention. The processor 11 can execute various functions of the urban management problem rapid reporting device based on the multimodal large model by running or executing the software program stored in the memory 12 and calling the data stored in the memory 12.

[0059] The communication interface 13 uses any transceiver or other device for communicating with other devices or communication networks, such as radio access networks (RAN), wireless local area networks (WLAN), etc. The communication interface 13 may include a receiving unit to implement a receiving function, and a sending unit to implement a sending function.

[0060] The communication bus 14 may include a pathway for transmitting information between the above-mentioned components.

[0061] It should be pointed out that Figure 6 The structure shown in the figure does not constitute a limitation on the rapid reporting device for urban management issues based on the multimodal large model, except Figure 6 In addition to the components shown, the device for quickly reporting urban management issues based on a multimodal large model may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0062] Example 1

[0063] The present invention provides a method for quickly reporting urban management issues based on a multi-modal large model. Figure 1-Figure 5 , including the steps of:

[0064] S1: The user uploads the event image to the cloud server through the client;

[0065] S2: cloud server stores event images;

[0066] S31: The recognition device recognizes that the event image is an urban management issue, and generates an event description item, which includes: event title, event level, event category, and event subcategory;

[0067] S4: The recognition device sends a request to the browser to call the big model, which is a model trained by combining text and image information. The big model includes: a multimodal language model, a machine learning model, and a deep learning model;

[0068] S5: The recognition device transmits the event picture and the event description item to the large model;

[0069] S6: The large model receives and parses the event image and event description item to obtain the information to be confirmed, which includes the event description information and event location information, including:

[0070] S61: The machine learning model and the deep learning model preprocess the event images, including: resizing the event images, color space conversion, and denoising;

[0071] S62: Machine learning models and deep learning models extract key elements of event images, including color and content of event images;

[0072] S63: The machine learning model and the deep learning model compare the key elements of the event image with the key elements of the preset images in the preset template library, and select the preset image whose similarity with the key elements of the event image is greater than a preset value;

[0073] S64: The machine learning model and the deep learning model read the preset event information of the corresponding preset image, wherein the preset event information includes: preset event level, preset event category and preset event subcategory; the preset event level includes: daily events, general events and major events; the preset event categories include: urban management, production safety and environmental sanitation; the preset event subcategories include: public health, urban environment, publicity and advertising and construction management;

[0074] S65: The machine learning model and the deep learning model traverse the event description items, write the preset event information into the corresponding event description items, and obtain the event description information;

[0075] S66: The machine learning model and the deep learning model read the event image to obtain event location information;

[0076] S67: The multimodal language model organizes the formats of the event description information and the event location information to obtain information to be confirmed;

[0077] S7: The multimodal language model sends the information to be confirmed to the client;

[0078] S8: The user checks the information to be confirmed and sends it to the urban management issue platform.

[0079] S9: The user checks the information to be confirmed. If it is incorrect, the user repeats the steps of uploading the event picture to the cloud server through the client and the subsequent steps;

[0080] S10: The user checks the information to be confirmed and cancels the report;

[0081] S32: The recognition device recognizes that the event image is not a city management issue, and sends a prompt to the client.

[0082] Specifically, the present invention provides a method for quickly reporting urban management issues based on a multimodal large model, comprising the steps of:

[0083] S1: The user uploads the event image to the cloud server through the client;

[0084] It should be noted that users upload the event pictures recorded by photos through the client. The event pictures can be taken by the watermark camera. The event pictures include watermarks showing the photo taking time, address, longitude and latitude, which facilitates subsequent identification and analysis of the pictures.

[0085] S2: cloud server stores event images;

[0086] It should be noted that the cloud server in this embodiment can be an object storage service (Object Storage Service, OSS), which is a massive, secure, low-cost, and highly reliable cloud storage service suitable for storing any type of files, with elastic expansion of capacity and processing power, and a variety of storage types to choose from, which fully optimizes storage costs. OSS has a platform-independent REST API interface, which can store and access any type of data in any application, at any time, and at any place; API, SDK package or OSS migration tool can be used to easily move massive data into or out of OSS. After the data is stored in OSS, standard storage (Standard) can be selected as the main storage method for mobile applications, large websites, picture sharing or hot audio and video, or low-frequency access storage (Infrequent Access), archive storage (Archive), cold archive storage (Cold Archive) or deep cold archive (Deep Cold Archive) with lower cost and longer storage period can be selected as the storage method for infrequently accessed data.

[0087] It should be noted that the cloud server stores event pictures, saves photos to the OSS server and returns a unique address for photo access. Here, the REST API interface provided by the OSS server is used for uploading.

[0088] It should be noted that REST API is the HTTP management interface provided by IM to the App backend. Its main purpose is to provide a backend management entrance for the App backend. For security reasons, REST API only provides HTTPS interface.

[0089] S31: Identify the event image as an urban management issue and generate event description items, which include: event title, event level, event category and event subcategory;

[0090] It should be noted that first, identify whether the event image is a problem in urban management. If it is, further describe the event attributes and generate event description items, namely event title, event level, event category and event subcategory. The above process can be responded to by json format in python, and ensure that the response result can be successfully loaded by json format in python, refer to Figure 2 For an event image, the generated event description items can be: "title": "event title", "event_level": "event level", "category_major": "event major category" and "category_minor": "event minor category";

[0091] S4: The recognition device sends a request to the browser to call the big model, which is a model trained by combining text and image information. The big model includes: a multimodal language model, a machine learning model, and a deep learning model;

[0092] It should be noted that the model obtained by combining text, image, video and audio information for deep learning training is a multimodal large model. This model can process multiple media data and achieve more intelligent information processing by learning the relationship between different modes. Specifically, the multimodal large model M6 is the world's largest AI pre-training model, with parameters jumping from trillions to 10 trillion, and has strong cognitive and innovative capabilities; Shengshu Technology's multimodal large model is committed to improving creativity and productivity; CLIP is a multimodal representation large model that can convert text, pictures, audio, video and other content into vector representations in high-dimensional space for multimodal reasoning of vision and language; Kunyuan is the world's first multimodal geographic science large model aimed at promoting the deep integration of geography and artificial intelligence; multimodal large models are widely used in natural language processing, computer vision, multimedia processing and other fields. As the scale of data expands, the performance and generalization capabilities of multimodal large models will be further improved, and the application scenarios will become more and more extensive.

[0093] It should be noted that, in this embodiment, the big model platform uses the big model service platform Tongyi Qianwen VL-Max, and sends the storage address of the event image obtained in step S2 and the event description item generated in step S31 to the big model through the API interface of Tongyi Qianwen VL-Max.

[0094] It should be noted that Tongyi Qianwen is an artificial intelligence language model. It has powerful functions such as document processing, audio and video understanding, intelligent coding, etc., and supports multimodal understanding, multi-language support, knowledge questions and answers, educational assistance, information retrieval and innovative applications. Especially in code writing, it is a large language model that specifically responds to human instructions. It can understand and answer questions in various fields, including common, complex and even rare questions; Tongyi Qianwen's functional features include: multi-round dialogue, which means that it can have continuous and coherent dialogue exchanges with users, understand and remember the context of the dialogue, and achieve in-depth communication; it can generate various articles, stories, news releases, slogans, product descriptions and other types of copywriting according to user needs; when faced with complex problems, it can perform a certain degree of logical analysis and reasoning to give reasonable answers; it can integrate and understand multiple information sources such as text and images, and conduct cross-modal intelligent interaction.

[0095] It should be noted that the large model includes: multimodal language model, machine learning model and deep learning model. The multimodal language model has the characteristics of multimodal fusion and can process various types of data such as text, images, audio, etc., and extract meaningful information from them; it can understand the relationship between different modalities, such as extracting descriptive information from images, or converting descriptive information into images; it can perform a variety of tasks, such as text generation, question and answer, translation, image description generation, etc.

[0096] It should be noted that machine learning models can achieve supervised learning, unsupervised learning, semi-supervised learning and reinforcement learning. Among them, supervised learning is training with labeled data, such as classification, regression and other tasks; unsupervised learning is learning patterns and structures from unlabeled data, such as clustering, dimensionality reduction and other tasks; semi-supervised learning is training with a small amount of labeled data and a large amount of unlabeled data; reinforcement learning is learning the optimal strategy by interacting with the environment.

[0097] It should be noted that deep learning models include: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), Transformer Architectures (Transformers), Generative Adversarial Networks (GANs) and Autoencoders (Autoencoders). Among them, Convolutional Neural Networks (CNNs) are used for image processing and recognition; Recurrent Neural Networks (RNNs) are used to process sequence data, such as text and speech; Long Short-Term Memory Networks (LSTMs) are an improved version of RNNs used to capture long-term dependencies; Transformer Architectures (Transformers) are used to efficiently process sequence data and are widely used in natural language processing; Generative Adversarial Networks (GANs) are used to generate realistic data samples; Autoencoders (Autoencoders) are used for feature learning and dimensionality reduction.

[0098] S5: The recognition device transmits the event picture and the event description item to the large model;

[0099] It should be noted that Tongyi Qianwen VL-Max provides an HTTP calling interface, which can send a POST request to the corresponding address, https: / / dashscope.aliyuncs.com / api / v1 / services / aigc / multimodal-generation / generation, through Python's requests module, and encapsulate the event image and event description items into the body parameter.

[0100] It should be noted that a key architectural improvement of Tongyi Qianwen VL-Max (Qwen2-VL) is the implementation of Naive Dynamic Resolution support. Unlike the previous generation model Qwen-VL, Qwen2-VL can process images of any resolution without splitting them into blocks, thus ensuring consistency between the model input and the inherent information of the image. This approach more closely mimics human visual perception, enabling the model to process images of any clarity or size.

[0101] S6: The large model receives and parses the event image and event description item to obtain the information to be confirmed; the information to be confirmed includes the event description information and the event location information, including:

[0102] S61: The machine learning model and the deep learning model preprocess the event images, including: resizing the event images, color space conversion, and denoising;

[0103] It should be noted that the preprocessing of event images includes: image loading and format conversion. The preprocessing process first needs to load the image into the program and ensure that the image format is suitable for subsequent processing. Common image formats include JPEG, PNG, BMP, etc. After loading, the image may need to be converted to a unified format, such as grayscale image or RGB image; resizing. In order to adapt to different model input requirements, it is usually necessary to adjust the image size. For example, many deep learning frameworks require the input image to have a fixed size. Common resizing methods include: scaling, cropping, padding, normalization, grayscale, denoising and edge detection. Among them, scaling (Scaling g) is to scale the image to the specified width and height; Cropping is to cut out a specified size area from the image, and Padding is to add pixels (usually black or white) to the edge of the image to make it reach the desired size; Normalization is to make the pixel values ​​between different images in the same range, usually normalizing the pixel values ​​to between 0 and 1; Grayscale Conversion, if the task does not require color information, the color image can be converted to a grayscale image, which can be achieved by calculating the average or weighted average of each pixel; Noise Removal Image denoising is to remove random noise in the image. A variety of filters can be used, such as median filtering, which replaces the central pixel value with the median value in the window; Gaussian filtering, which uses the weights of Gaussian distribution to perform weighted averaging on the pixel values ​​in the window; histogram equalization, which can enhance the image contrast and make the image details more obvious, is achieved by redistributing the intensity values ​​of the image; edge detection is used to highlight the boundary information in the image; feature standardization is to eliminate the scale differences between different features, usually converting the feature values ​​into the form of zero mean and unit variance.

[0104] S62: Machine learning models and deep learning models extract key elements of event images, including color and content of event images;

[0105] S63: The machine learning model and the deep learning model compare the key elements of the event image with the key elements of the preset images in the preset template library, and select the preset image whose similarity with the key elements of the event image is greater than a preset value;

[0106] S64: The machine learning model and the deep learning model read the preset event information of the corresponding preset image, wherein the preset event information includes: preset event level, preset event category and preset event subcategory; the preset event level includes: daily events, general events and major events; the preset event categories include: urban management, production safety and environmental sanitation; the preset event subcategories include: public health, urban environment, publicity and advertising and construction management;

[0107] It should be noted that the preset template library contains a variety of image information, including images corresponding to urban management issues. By comparing the event image with the preset template library, the uploaded event image is analyzed through a deep learning and trained large model, and a preset image with a similarity with the event image greater than a preset value is selected. The preset value can be set according to usage needs;

[0108] S65: The machine learning model and the deep learning model traverse the event description items, write the preset event information into the corresponding event description items, and obtain the event description information;

[0109] S66: The machine learning model and the deep learning model read the event image to obtain event location information;

[0110] It should be noted that the preset event information of the corresponding preset picture, that is, the specific content of the event description item, is read;

[0111] It should be noted that the event description items are parsed in json format in python. Figure 2 The event picture shown in the figure traverses the event description items, writes the preset event information into the corresponding event description items, and the obtained event description information can be: event title "title": "broken manhole cover", event level "event_level": "general event", event category "category_major": "city management", event subcategory "category_minor": "city appearance and environment";

[0112] It should be noted that machine learning models and deep learning models have the function of image recognition and can read Figure 2After viewing the event picture, we can get the event location information: "Event description": "The manhole cover is seriously damaged, posing a safety hazard and needs to be repaired or replaced in time." "Address": "Minghu International Information Innovation Industrial Park, Jinan City", "Longitude": "117.0468E", "Latitude": "36.6621N".

[0113] S67: The multimodal language model organizes the formats of the event description information and the event location information to obtain information to be confirmed;

[0114] It should be noted that the multimodal language model integrates event description information and event location information into a JSON file; you can follow the following steps: Define a dictionary to store all the information. Use the json module in Python to convert the dictionary into a JSON string.

[0115] S7: The multimodal language model sends the information to be confirmed to the client;

[0116] It should be noted that the multimodal language model automatically fills the information to be confirmed into the preset form, and the preset form is displayed in real time on the user client, and the client can check whether the information is accurate.

[0117] S8: The user checks the information to be confirmed and sends it to the urban management problem platform 5.

[0118] It should be noted that users can check the information to be confirmed through the preset form automatically filled in by the large model. If there is any error, they can directly modify or cancel it. If there is no error, they can click Submit to complete the reporting process. The user checking process can improve the accuracy of problem identification in the automatic reporting process and avoid reporting errors. It can also avoid errors caused by manual filling and improve the efficiency of reporting urban management issues.

[0119] S9: The user checks the information to be confirmed. If it is incorrect, the user repeats the steps of uploading the event picture to the cloud server through the client and the subsequent steps.

[0120] It should be noted that users can check the information to be confirmed through the preset form automatically filled in by the large model. If there is any error, they can directly modify or cancel it, or upload it again. If there is no error, they can click Submit to complete the reporting process. The user checking process can improve the accuracy of problem identification in the automatic reporting process and avoid reporting errors. It can also avoid errors caused by manual filling and improve the efficiency of reporting urban management issues.

[0121] S10: The user checks the information to be confirmed and cancels the report.

[0122] It should be noted that in the above-mentioned method for quickly reporting urban management problems based on a multimodal large model, the user can cancel the problem reporting process through the client.

[0123] S32: The recognition device recognizes that the event image is not a city management issue, and sends a prompt to the client;

[0124] It should be noted that identifying event images stored in the cloud server is not an urban management issue. The client prompts the user to re-upload the images to avoid problems being discovered during subsequent processing and delaying the time limit.

[0125] It can be understood that the present invention provides a method for quickly reporting urban management problems based on a multimodal big model, including the following steps: a user uploads an event picture to a cloud server through a client; the convenience of problem reporting is improved; the cloud server stores the event picture and generates a storage address; the event picture is identified as an urban management problem and an event description item is generated; the big model is called and the multimodal big model is used to further accurately analyze the urban management problem and improve the accuracy of problem identification; the event picture and the event description item are transmitted to the big model, the big model receives and parses the event picture and the event description item to obtain information to be confirmed; the big model reports the information to be confirmed to the client, and the user checks the information to be confirmed. If it is submitted correctly with the uploaded event picture information, the method integrates advanced technologies such as image recognition, natural language processing and deep learning, and can efficiently extract and identify relevant event information from photos, improve the accuracy of problem identification and speed up efficiency. The problem information in the event picture is accurately extracted, and the big model will automatically fill in the information to be confirmed. After receiving the automatically generated form, the user can complete the reporting process by simply checking and confirming, which reduces the links of manual intervention and data entry, reduces the probability of errors, and effectively improves the speed and efficiency of event reporting.

[0126] Example 2

[0127] This embodiment describes the event location information in detail based on the first embodiment. The event location information includes event description, event address, longitude and latitude.

[0128] It should be noted that the event description information includes the event title, event level, event category and event subcategory, event description, event address, longitude and latitude; the event title is obtained from the identified event image, and the specific content in the identified event image is referenced. Figure 2The event picture, the event title can be a damaged manhole cover, the event level can be based on the event picture and can be selected from the preset event levels, which can be a general event; the event category can be based on the event picture and can be selected from the preset event categories, which can be urban management; the event subcategory can be based on the event picture and can be selected from the preset event subcategories, which can be urban environment; the event description summarizes the recognition results of the event picture and is described as: the manhole cover is seriously damaged, posing a safety hazard and needs to be repaired or replaced in time to facilitate subsequent processing after the report is successful; the address, longitude and latitude can be read from the watermark in the event picture.

[0129] Example 3

[0130] This embodiment describes the confirmation information in detail based on implementation 2: Figure 1-Figure 5 The client includes a preset form, and the multimodal language model large model sends the information to be confirmed to the client, including: S71: the multimodal language model large model establishes a connection with the client; S72: the multimodal language model large model fills the information to be confirmed into the preset form.

[0131] It should be noted that S7: The multimodal language model sends the information to be confirmed to the client, that is, automatically fills the information to be confirmed into the preset form of the client, refer to Figure 3 The preset form content is the information to be confirmed, including event description information and event location information. This process is the process of big model interaction with front-end client and back-end parsing, which is convenient for users to confirm and submit information. By analyzing the on-site problem photos uploaded by users, the big model technology is used to quickly parse and identify problem information, and automatically fill in the preset form. Users only need to conduct a final check to complete the event sending, which improves sending efficiency and avoids errors in manual filling.

[0132] Example 4

[0133] The present invention also provides a device for quickly reporting urban management problems based on a multimodal large model, comprising: a processor 11 and a memory 12, the memory 12 is used to store computer program code, the computer program code includes computer instructions, and when the processor 11 executes the computer instructions, the device for quickly reporting urban management problems based on a multimodal large model is used to execute a method for quickly reporting urban management problems based on a multimodal large model, any one of the method processes shown in the above method embodiments.

[0134] Example 5

[0135] The present invention also provides an electronic device, comprising: a processor 11, a storage medium and a computer program, wherein the computer program is stored in the storage medium, and is characterized in that when the computer program is executed by the processor, a method for quickly reporting urban management problems based on a multimodal large model, any one of the method flows shown in the above method embodiments, is implemented.

[0136] Example 6

[0137] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for quickly reporting urban management problems based on a multimodal large model, any one of the method flows shown in the above method embodiments, is implemented.

[0138] It can be seen from the above embodiments that the method for quickly reporting urban management issues based on a multimodal large model provided by the present invention achieves at least the following beneficial effects:

[0139] The invention provides a method for quickly reporting urban management problems based on a multimodal big model, comprising the following steps: a user uploads an event picture to a cloud server through a client; the convenience of problem reporting is improved; the cloud server stores the event picture and generates a storage address; the event picture is identified as an urban management problem and an event description item is generated; the big model is called and the multimodal big model is used to further accurately analyze the urban management problem and improve the accuracy of problem identification; the event picture and the event description item are transmitted to the big model, the big model receives and parses the event picture and the event description item to obtain information to be confirmed; the big model reports the information to be confirmed to the client, the user checks the information to be confirmed, and if it is correctly submitted with the uploaded event picture information, the method integrates advanced technologies such as image recognition, natural language processing and deep learning, can efficiently extract and identify relevant event information from photos, improve the accuracy of problem identification and speed up efficiency, the problem information in the event picture is accurately extracted, the big model automatically fills in the information to be confirmed, and the user completes the reporting process by simply checking and confirming after receiving the automatically generated form, reducing the links of manual intervention and data entry, reducing the probability of errors, and effectively improving the speed and efficiency of event reporting.

[0140] Although some specific embodiments of the present invention have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present invention.

[0141] It will be appreciated by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for quickly reporting urban management issues based on a multimodal large model, characterized in that: Includes steps: Users upload event pictures to the cloud server through the client; The cloud server stores the event picture; The recognition device recognizes that the event image is an urban management issue and generates an event description item, wherein the event description item includes: event title, event level, event major category and event minor category; The recognition device sends a request to the browser to call a large model, wherein the large model is a model obtained by training the text and image information together, and the large model includes: a multimodal language model, a machine learning model, and a deep learning model; The recognition device transmits the event picture and the event description item to the large model; The large model receives and parses the event image and the event description item to obtain information to be confirmed, wherein the information to be confirmed includes event description information and event location information, including: The machine learning model and the deep learning model preprocess the event picture, wherein the preprocessing includes: adjusting the size of the event picture, color space conversion, and denoising; The machine learning model and the deep learning model extract key elements of the event picture, wherein the key elements include color and content of the event picture; The machine learning model and the deep learning model compare the key elements of the event picture with the key elements of the preset pictures in the preset template library, and select the preset picture whose similarity with the key elements of the event picture is greater than a preset value; The machine learning model and the deep learning model read the preset event information of the corresponding preset picture, wherein the preset event information includes: preset event level, preset event category and preset event subcategory; the preset event level includes: daily events, general events and major events; the preset event categories include: urban management, production safety and environmental sanitation; the preset event subcategories include public health, urban environment, publicity and advertising and construction management; The machine learning model and the deep learning model traverse the event description items, write the preset event information into the corresponding event description items, and obtain event description information; The machine learning model and the deep learning model read the event picture to obtain event location information; The multimodal language model organizes the formats of the event description information and the event location information to obtain the information to be confirmed; The multimodal language model sends the information to be confirmed to the client; The user checks the information to be confirmed and sends it to the city management problem platform; The user checks the information to be confirmed, and if it is incorrect, repeats the steps of uploading the event picture to the cloud server through the client and thereafter; The user checks the information to be confirmed and cancels the report; If the recognition device recognizes that the event image is not a city management issue, a prompt is sent to the client.

2. The method for quickly reporting urban management issues based on a multimodal large model according to claim 1 is characterized in that: The event location information includes event description, event address, longitude and latitude.

3. The method for quickly reporting urban management issues based on a multimodal large model according to claim 1 is characterized in that: The client includes a preset form, and the multimodal language model sends the information to be confirmed to the client, including: the multimodal language model establishes a connection with the client; and the multimodal language model fills the information to be confirmed into the preset form.

4. A device for quickly reporting urban management issues based on a multimodal large model, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the device for quickly reporting urban management issues based on a multimodal large model executes the method for quickly reporting urban management issues based on a multimodal large model as described in any one of claims 1-3.

5. An electronic device comprising: A processor, a storage medium and a computer program, wherein the computer program is stored in the storage medium, and is characterized in that when the computer program is executed by the processor, the method for quickly reporting urban management problems based on a multimodal large model as described in any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for quickly reporting urban management issues based on a multimodal large model as described in any one of claims 1 to 3 is implemented.

Images