Method for constructing urban governance multi-modal large model based on problem attributes
By constructing a multimodal model of urban governance based on problem attributes, the problem that qualitative description cannot be given in the existing technology is solved, and the qualitative description ability of the urban governance model when outputting target information is realized, and the precise processing of urban management is supported.
Patent Information
- Application Number
- CN202510365954.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-26
- Publication Date
- 2025-08-12
AI Technical Summary
The existing multimodal model of urban governance cannot provide a clear qualitative description while outputting target information, especially the severity and frequency of problems in the picture scenario.
Build a multimodal model of urban governance based on problem attributes. By obtaining the training set and training the multimodal model of urban governance, use components such as visual feature extractor, projector, language model word segmenter, transformation module and fully connected network to map and splice features to generate qualitative descriptions of the categories, severity and frequency of violation scenarios.
It is achieved that while outputting urban management target information, a clear qualitative description can be given, which facilitates the handling of violation scenarios, severity and frequency in urban management.
Smart Images
Figure CN120472293A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of smart city technology, and more specifically, to a method for constructing a large multimodal model of urban governance based on problem attributes. Background Art
[0002] Convolutional neural networks have been widely used in the field of computer vision. Object detection involves identifying the location of objects in an image and annotating them with bounding boxes. Using convolutional neural networks for object detection can obtain target labels and location information, but lacks detailed information descriptions.
[0003] Currently, the multimodal big model for urban governance can output target information and provide detailed scene descriptions, but it cannot output clear qualitative descriptions. For example, the text output by the multimodal big model for urban governance is "This image shows a street scene, focusing mainly on a row of mobile vendor carts parked on the roadside. These vehicles look like snack stalls converted from three-wheeled electric vehicles, with red awnings and billboards displaying information about the food they sell. Several vendors can be seen standing by their stalls, seemingly preparing ingredients or waiting for customers. There are some buildings and trees in the background, indicating that this is an outdoor urban environment. The sky is partially covered by clouds and the light is dim, possibly in the early morning or evening. Overall, this scene reflects the common small vendor activities on urban streets." This shows that the current multimodal big model for urban governance only describes the scene in the image, and cannot provide a qualitative description of the problems in the image scene, nor can it determine the severity of the problems in the image scene. Therefore, how to provide urban management target information while providing a clear qualitative description of it is an urgent problem that needs to be solved in urban management. Summary of the Invention
[0004] The technical problem to be solved by this application is: how to output urban management target information while giving it a clear qualitative description.
[0005] The technical solution adopted by this application to solve its technical problems is:
[0006] In the first aspect, the present application provides a method for constructing a multimodal large model of urban governance based on problem attributes, the method comprising the following steps:
[0007] S1, obtain the training set;
[0008] S2, building a large multimodal model of urban governance;
[0009] S3, training the urban governance multimodal large model in step S2 using the training set obtained in step S1;
[0010] S4, obtain the trained multimodal large model of urban governance.
[0011] Based on the first aspect, further, the training set includes: a first picture, an instruction text corresponding to the first picture, and an answer text corresponding to the first picture and the instruction text.
[0012] Based on the first aspect, further, the answer text includes a first question attribute text.
[0013] Based on the first aspect, further, the first question attribute text includes: category, severity and frequency of the violation scenario.
[0014] Based on the first aspect, further, the urban governance multimodal large model in step S2 includes: a visual feature extractor, a projector, a first large language model segmenter, a first large language model embedding module, a first conversion module, a first single-layer fully connected network, a learnable Q matrix creation module, a second conversion module, a splicing module, and a first large language model encoding module; when the urban governance multimodal large model in step S2 is trained with the training set obtained in step S1, the first picture in the training set is input into the visual feature extractor, the output of the visual feature extractor is input into the projector, and the projector outputs the picture embedding feature; the instruction text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the instruction text tag; the instruction text tag is input into the first large language model embedding module, and the first large language model embedding module outputs the instruction text embedding feature; the answer text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the answer text tag; the answer text tag is input into the first large language model embedding module, and the first large language model embedding module outputs the answer text embedding feature; the answer text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the answer text tag; The first question attribute text is input into the first conversion module, and the first conversion module outputs the first question matrix; the first question matrix output by the first conversion module is input into the first single-layer fully connected network, and the first single-layer fully connected network outputs the KV matrix; the learnable Q matrix creation module creates a learnable Q matrix; the KV matrix output by the first single-layer fully connected network is input into the second conversion module, and the learnable Q matrix output by the learnable Q matrix creation module is input into the second conversion module, and the second conversion module outputs attribute features; the image embedding features output by the projector, the instruction text embedding features output by the first large language model embedding module, and the attribute features output by the second conversion module are input into the splicing module, and the splicing module outputs the first feature; the first feature output by the splicing module is input into the first large language model encoding module, and the first large language model encoding module outputs the predicted text embedding features; the predicted text embedding features output by the first large language model encoding module are input into the first large language model word segmenter, and the predicted text is obtained by decoding by the first large language model word segmenter; the second question attribute text in the predicted text output by the first large language model word segmenter is input into the first conversion module, and the first conversion module outputs the second question matrix.
[0015] The predicted text includes the second question attribute text.
[0016] The attribute text of the second question includes: the category, severity and frequency of the violation scenario.
[0017] The first conversion module may map the first question attribute text of the answer text into a first question matrix. The first conversion module may map the second question attribute text of the prediction text into a second question matrix.
[0018] Based on the first aspect, further, the specific method of the first conversion module mapping the first question attribute text of the answer text to the first question matrix includes: the severity label adopts a one-hot encoding scheme, the quantization value of the label "serious" is 100, the quantization value of the label "general" is 010, and the quantization value of the label "mild" is 001; the frequency label adopts a one-hot encoding scheme, the quantization value of the label "high incidence" is 100, the quantization value of the label "common" is 010, and the quantization value of the label "rare" is 001.
[0019] Based on the first aspect, further, the specific method of the first conversion module mapping the second question attribute text of the predicted text into the second question matrix includes: the severity label adopts a one-hot encoding scheme, the quantization value of the label "serious" is 100, the quantization value of the label "general" is 010, and the quantization value of the label "mild" is 001; the frequency label adopts a one-hot encoding scheme, the quantization value of the label "high incidence" is 100, the quantization value of the label "common" is 010, and the quantization value of the label "rare" is 001.
[0020] Based on the first aspect, further, the initial values of all elements of the first problem matrix are all 0; the first problem matrix has four dimensions, the first dimension corresponds to different samples, the second dimension corresponds to different violation scene category classification labels, the third dimension is the severity label quantization value, and the fourth dimension is the frequency label quantization value.
[0021] Based on the first aspect, further, the training set obtained through step S1 for training the urban governance multimodal large model in step S2 includes first-stage training and second-stage training.
[0022] Based on the first aspect, further, the loss function used in the first stage training is a first loss function:
[0023]
[0024] in, Represents the total loss during the training phase; represents the text space cross entropy loss; Represents the attribute classification cross entropy loss; α represents the adjustment coefficient of the attribute classification cross entropy loss influence;
[0025] Among them, the text space cross entropy loss The calculation formula is:
[0026]
[0027] ∩ represents the probability distribution cross entropy loss calculation function; Represents the predicted text embedding features output by the first language model encoding module; Represents the answer text embedding features output by the first language model embedding module;
[0028] Among them, attribute classification cross entropy loss The calculation formula is:
[0029]
[0030] in, represents the second problem matrix; Denotes the first problem matrix.
[0031] Based on the first aspect, further, the second stage training adopts the first loss function as the loss function, and also adopts the problem matrix similarity measure to evaluate the urban governance multimodal large model, and takes the parameters of the urban governance multimodal large model with the largest problem matrix similarity measure as the parameters of the trained urban governance multimodal large model; wherein, the formula of the problem matrix similarity measure is:
[0032]
[0033] Among them, Sim represents the similarity measure of the problem matrix; Min represents the minimum value; Max represents the maximum value; Ham represents the Hamming distance calculation function; Mat Train Represents the components of different matrices composed of the first dimension in the first problem matrix.
[0034] In a second aspect, the present application also provides an electronic device comprising a memory and a processor; the memory is used to store one or more programs; when the one or more programs are executed by the processor, a method as described in any one of the first aspects above is implemented.
[0035] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements any one of the methods in the first aspect when executed by a processor.
[0036] The beneficial effects of this application are:
[0037] The multimodal large model of urban governance constructed by the method of constructing a multimodal large model of urban governance based on problem attributes in this application can output urban management target information while giving it a clear qualitative description, so as to handle relevant matters in urban management according to the violation scenarios, severity and frequency in the qualitative description. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A flowchart of a method for constructing a multimodal large model of urban governance based on problem attributes provided in an embodiment of the present application.
[0039] Figure 2 This is an example of the first picture provided in the embodiment of the present application.
[0040] Figure 3 This is another example of the first picture provided in the embodiment of the present application.
[0041] Figure 4 A schematic diagram of the first question matrix provided in an embodiment of the present application.
[0042] Figure 5 This is a structural block diagram of the electronic device provided in an embodiment of the present application.
[0043] The reference numerals in the drawings are: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION
[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.
[0045] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.
[0046] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.
[0047] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.
[0048] In the description of the embodiments of the present application, "a plurality of" means at least 2.
[0049] First, as Figure 1 As shown, an embodiment of the present application provides a method for constructing a multimodal large model of urban governance based on problem attributes, the method comprising the following steps:
[0050] S1, obtain the training set;
[0051] S2, building a large multimodal model of urban governance;
[0052] S3, training the urban governance multimodal large model in step S2 using the training set obtained in step S1;
[0053] S4, obtain the trained multimodal large model of urban governance.
[0054] The trained urban governance multimodal big model is a multimodal big model of urban governance constructed by a method for constructing an urban governance multimodal big model based on problem attributes.
[0055] In one embodiment of the present application, the training set includes: a first picture, an instruction text corresponding to the first picture, and an answer text corresponding to the first picture and the instruction text.
[0056] In one embodiment of the present application, the first picture is a picture of a scene of urban governance violations.
[0057] In one embodiment of the present application, the answer text corresponding to the first picture and the instruction text includes the first question attribute text.
[0058] In one embodiment of the present application, the first question attribute text includes: the category, severity, and frequency of the violation scenario.
[0059] The categories of illegal scenes may include: shared bicycles lying down, vendors' tricycles, vendors' four-wheeled trucks, occupying the road for business, road waterlogging, scattered garbage, abandoned furniture and equipment, illegal street slopes, outdoor advertisements, items drying in the sun, floor signs set up in squares, slogan-type propaganda materials, peddlers in carports, occupying the road for business, packaged garbage, overflowing garbage, scattered garbage, cracked and damaged, construction waste materials, traffic signs, waste collection, poultry and livestock, green guardrails, missing manhole covers, and disorderly Piled materials, graffiti, propaganda banners, inflatable models or arches, unclosed transfer box doors, missing transfer box bottom covers, unclosed facility box doors, overturned crash barrels, overturned sidewalk piles, fallen guardrails, roadside renovations, medical waste, damaged barrels, missing street trees, abnormal fire hydrants, missing pole bottom covers, food waste, tree protection pools, overturned isolation bodies, damaged sidewalks, curbstones, construction site fencing, street chairs, spilled soil, missing grates, and overturned street trees.
[0060] Severity levels can include severe, moderate, and mild.
[0061] Frequency levels can include frequent, common, and rare.
[0062] In one embodiment of the present application, the first picture is as follows Figure 2 As shown. The instruction text corresponding to the first image is "Describe the urban management issues in the image." The answer text corresponding to the first image and instruction text is "There is a shared bicycle overturned in the image [[315, 623, 760, 950]], which is a minor common problem." Here, 315 is the horizontal coordinate of the upper left corner of the target positioning box, 623 is the vertical coordinate of the upper left corner of the target positioning box, 760 is the horizontal coordinate of the lower right corner of the target positioning box, and 950 is the vertical coordinate of the lower right corner of the target positioning box. It should be noted that there is no target positioning box in the first image. The horizontal coordinates of the upper left corner of the target positioning box, the vertical coordinates of the upper left corner of the target positioning box, the horizontal coordinates of the lower right corner of the target positioning box, and the vertical coordinates of the lower right corner of the target positioning box in the answer text are obtained through manual annotation. The attribute text of the first question is "shared bicycle overturned, minor, common"; where "shared bicycle overturned" is the category of the violation scenario, "minor" is the severity, and "common" is the frequency.
[0063] In another embodiment of the present application, the first picture is as follows: Figure 3As shown. The instruction text corresponding to the first picture is "Describe the urban management problems in the picture." The answer text corresponding to the first picture and the instruction text is "There are shared bicycles lying down in the picture [[117,415,435,735]], shared bicycles lying down [[285,577,765,845]], shared bicycles lying down [[5,286,290,502]] and shared bicycles lying down [[10,410,260,720]], which are general common problems." Among them, 117 is the horizontal coordinate of the upper left corner of the first target positioning box, 415 is the vertical coordinate of the upper left corner of the first target positioning box, 435 is the horizontal coordinate of the lower right corner of the first target positioning box, and 735 is the vertical coordinate of the lower right corner of the first target positioning box. The vertical coordinate of the top left corner of the second target positioning frame is 285, the vertical coordinate of the top left corner of the second target positioning frame is 577, the horizontal coordinate of the bottom right corner of the second target positioning frame is 765, and the vertical coordinate of the bottom right corner of the second target positioning frame is 845. The horizontal coordinate of the top left corner of the third target positioning frame is 5, the vertical coordinate of the top left corner of the third target positioning frame is 286, the horizontal coordinate of the bottom right corner of the third target positioning frame is 290, and the vertical coordinate of the bottom right corner of the third target positioning frame is 502. The horizontal coordinate of the top left corner of the fourth target positioning frame is 10, the vertical coordinate of the top left corner of the fourth target positioning frame is 410, the horizontal coordinate of the bottom right corner of the fourth target positioning frame is 260, and the vertical coordinate of the bottom right corner of the fourth target positioning frame is 720. It should be noted that the first target positioning frame, the second target positioning frame, the third target positioning frame, and the fourth target positioning frame do not exist in the first image. The horizontal coordinate of the upper left corner of the first target positioning box, the vertical coordinate of the upper left corner of the first target positioning box, the horizontal coordinate of the lower right corner of the first target positioning box, the vertical coordinate of the lower right corner of the first target positioning box, the horizontal coordinate of the upper left corner of the second target positioning box, the vertical coordinate of the upper left corner of the second target positioning box, the horizontal coordinate of the lower right corner of the second target positioning box, the vertical coordinate of the lower right corner of the second target positioning box, the horizontal coordinate of the upper left corner of the third target positioning box, the vertical coordinate of the upper left corner of the third target positioning box, the horizontal coordinate of the lower right corner of the third target positioning box, the vertical coordinate of the lower right corner of the third target positioning box, the horizontal coordinate of the upper left corner of the fourth target positioning box, the vertical coordinate of the upper left corner of the fourth target positioning box, the horizontal coordinate of the lower right corner of the fourth target positioning box, and the vertical coordinate of the lower right corner of the fourth target positioning box in the answer text are obtained through manual annotation. The attribute text of the first question is "shared bicycle overturned, general, common"; among them, "shared bicycle overturned" is the category of the violation scenario, "general" is the severity, and "common" is the frequency.
[0064] In one embodiment of the present application, the urban governance multimodal large model in step S2 includes: a visual feature extractor, a projector, a first large language model segmenter, a first large language model embedding module, a first conversion module, a first single-layer fully connected network, a learnable Q matrix creation module, a second conversion module, a splicing module, and a first large language model encoding module; when the urban governance multimodal large model in step S2 is trained with the training set obtained in step S1, the first picture in the training set is input into the visual feature extractor, the output of the visual feature extractor is input into the projector, and the projector outputs the picture embedding feature; the instruction text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the instruction text tag; the instruction text tag is input into the first large language model embedding module, and the first large language model embedding module outputs the instruction text embedding feature; the answer text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the answer text tag; the answer text tag is input into the first large language model embedding module, and the first large language model embedding module outputs the answer text embedding feature; the answer text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the answer text tag; the answer text tag is input into the first large language model embedding module, and the first large language model embedding module outputs the answer text embedding feature; The first question attribute text is input into the first conversion module, and the first conversion module outputs the first question matrix; the first question matrix output by the first conversion module is input into the first single-layer fully connected network, and the first single-layer fully connected network outputs the KV matrix; the learnable Q matrix creation module creates a learnable Q matrix; the KV matrix output by the first single-layer fully connected network is input into the second conversion module, and the learnable Q matrix output by the learnable Q matrix creation module is input into the second conversion module, and the second conversion module outputs attribute features; the image embedding features output by the projector, the instruction text embedding features output by the first large language model embedding module, and the attribute features output by the second conversion module are input into the splicing module, and the splicing module outputs the first feature; the first feature output by the splicing module is input into the first large language model encoding module, and the first large language model encoding module outputs the predicted text embedding features; the predicted text embedding features output by the first large language model encoding module are input into the first large language model word segmenter, and the predicted text is obtained by decoding by the first large language model word segmenter; the second question attribute text in the predicted text output by the first large language model word segmenter is input into the first conversion module, and the first conversion module outputs the second question matrix.
[0065] In one embodiment of the present application, the visual feature extractor is a Vision Transformer, which can be either ViT-bigG or EVA2-CLIP-E. The Vision Transformer is a model that introduces the Transformer architecture into the field of computer vision. The visual feature extractor can divide an image into multiple small blocks and then convert these small blocks into vector sequences. The visual feature extractor then processes these vector sequences using a Transformer (a deep learning model) encoder to extract high-level features of the image.
[0066] In one embodiment of the present application, the projector can be a two-layer MLP (i.e., multi-layer perceptron) network or a cross-attention network. The projector can convert data from different sources or different modalities into a text feature space.
[0067] The first language model includes a first language model word segmenter, a first language model embedding module, and a first language model encoding module. In one embodiment of the present application, the first language model can be Qwen or ChatGLM.
[0068] The predicted text includes the second question attribute text. The second question attribute text includes: the category, severity and frequency of the violation scenario.
[0069] The first question attribute text refers to the question attribute text in the answer text.
[0070] The second question attribute text refers to the question attribute text in the prediction text.
[0071] The first conversion module may map the first question attribute text into a first question matrix. The first conversion module may map the second question attribute text into a second question matrix.
[0072] The first single-layer fully connected network is used to convert the first problem matrix into a KV matrix, where the first half of the KV matrix is the K (Key) matrix and the second half of the KV matrix is the V (Value) matrix.
[0073] The learnable Q matrix creation module is used to create a learnable Q matrix.
[0074] The second conversion module is used to calculate the cross-attention output and convert the input KV matrix and learnable Q matrix into attribute features.
[0075] The splicing module is used to splice the image embedding feature, the instruction text embedding feature, and the attribute feature into a first feature. In one embodiment of the present application, the splicing module splices the image embedding feature, the instruction text embedding feature, and the attribute feature into the first feature in a specific method including: sequentially connecting the image embedding feature, the instruction text embedding feature, and the attribute feature to obtain the first feature.
[0076] Image embedding features refer to the feature vectors used to represent images.
[0077] The instruction text markup refers to a series of "tokens" corresponding to the instruction text. These tokens can be words, subwords, characters, or a combination thereof.
[0078] In one embodiment of the present application, the instruction text tag is obtained by inputting the instruction text into a first large language model word segmenter, and the first large language model word segmenter decomposing the instruction text.
[0079] The instruction text embedding feature refers to the feature vector used to represent the instruction text.
[0080] Answer text tags refer to a series of "tokens" corresponding to the answer text.
[0081] In one embodiment of the present application, the answer text tag is obtained by inputting the answer text into the first large language model word segmenter, and the first large language model word segmenter decomposing the answer text.
[0082] Answer text embedding features refer to the feature vectors used to represent answer texts.
[0083] Predicted text embedding features refer to the feature vectors used to represent predicted text.
[0084] In one embodiment of the present application, the specific method by which the first conversion module maps the first question attribute text of the answer text to the first question matrix includes: the severity label adopts a one-hot encoding scheme, the quantization value of the label "serious" is 100, the quantization value of the label "general" is 010, and the quantization value of the label "mild" is 001; the frequency label adopts a one-hot encoding scheme, the quantization value of the label "high incidence" is 100, the quantization value of the label "common" is 010, and the quantization value of the label "rare" is 001.
[0085] In one embodiment of the present application, the specific method by which the first conversion module maps the second question attribute text of the predicted text into the second question matrix includes: the severity label adopts a one-hot encoding scheme, the quantization value of the label "severe" is 100, the quantization value of the label "general" is 010, and the quantization value of the label "mild" is 001; the frequency label adopts a one-hot encoding scheme, the quantization value of the label "high incidence" is 100, the quantization value of the label "common" is 010, and the quantization value of the label "rare" is 001.
[0086] like Figure 4 As shown, in one embodiment of the present application, the initial values of all elements of the first problem matrix are all 0; the first problem matrix has four dimensions, the first dimension corresponds to different samples, the second dimension corresponds to different violation scene category classification labels, the third dimension is the severity label quantization value, and the fourth dimension is the frequency label quantization value.
[0087] In one embodiment of the present application, training the urban governance multimodal large model in step S2 using the training set obtained in step S1 includes first-stage training and second-stage training.
[0088] During the first stage of training, the first language model encoding module and visual feature extractor are frozen, and the projector and the second conversion module are trained.
[0089] During the second stage of training, the visual feature extractor, projector and second conversion module are frozen, and the first language model encoding module is trained.
[0090] In one embodiment of the present application, the loss function used in the first stage of training is the first loss function:
[0091]
[0092] in, Represents the total loss during the training phase; represents the text space cross entropy loss; Represents the attribute classification cross entropy loss; α represents the adjustment coefficient of the attribute classification cross entropy loss influence;
[0093] Among them, the text space cross entropy loss The calculation formula is:
[0094]
[0095] ∩ represents the probability distribution cross entropy loss calculation function; Represents the predicted text embedding features output by the first language model encoding module; Represents the answer text embedding features output by the first language model embedding module;
[0096] Among them, attribute classification cross entropy loss The calculation formula is:
[0097]
[0098] in, represents the second problem matrix; Denotes the first problem matrix.
[0099] In one embodiment of the present application, the second-stage training uses the first loss function as the loss function, and also uses the problem matrix similarity measure to evaluate the urban governance multimodal large model. The parameters of the urban governance multimodal large model with the largest value of the problem matrix similarity measure are used as the parameters of the trained urban governance multimodal large model; wherein the formula of the problem matrix similarity measure is:
[0100]
[0101] Among them, Sim represents the similarity measure of the problem matrix; Min represents the minimum value; Max represents the maximum value; Ham represents the Hamming distance calculation function; Mat Train Represents the components of different matrices composed of the first dimension in the first problem matrix.
[0102] Second, as Figure 5 As shown, an embodiment of the present application also provides an electronic device, which includes a memory 101 and a processor 102; the memory 101 is used to store one or more programs; when one or more programs are executed by the processor 102, a method as described in any one of the first aspects above is implemented.
[0103] The electronic device may further include a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines. The memory 101 may be used to store software programs and modules, and the processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 may be used to communicate signaling or data with other node devices.
[0104] Among them, the memory 101 can be but is not limited to: Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.
[0105] The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0106] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by the processor 102, implements a method as described in any one of the first aspects above. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0107] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.
[0108] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.
Claims
1. A method for constructing a large multimodal model of urban governance based on problem attributes, characterized by: The following steps are involved: S1, obtain the training set; S2, building a large multimodal model of urban governance; S3, training the urban governance multimodal large model in step S2 using the training set obtained in step S1; S4, obtain the trained multimodal large model of urban governance.
2. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 1 is characterized in that: The training set includes: a first picture, an instruction text corresponding to the first picture, and an answer text corresponding to the first picture and the instruction text.
3. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 2 is characterized in that: The answer text includes a first question attribute text.
4. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 3 is characterized in that: The first question attribute text includes: the category, severity and frequency of the violation scenario.
5. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 1 is characterized in that: The urban governance multimodal large model in step S2 includes: a visual feature extractor, a projector, a first large language model segmenter, a first large language model embedding module, a first conversion module, a first single-layer fully connected network, a learnable Q matrix creation module, a second conversion module, a splicing module, and a first large language model encoding module; when the urban governance multimodal large model in step S2 is trained with the training set obtained in step S1, the first picture in the training set is input into the visual feature extractor, the output of the visual feature extractor is input into the projector, and the projector outputs the picture embedding feature; the instruction text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the instruction text tag; the instruction text tag is input into the first large language model embedding module, and the first large language model embedding module outputs the instruction text embedding feature; the answer text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the answer text tag; the answer text tag is input into the first large language model embedding module, and the first large language model embedding module outputs the answer text embedding feature; the first question attribute of the answer text in the training set is input into the first large language model segmenter, and the first large language model segmenter outputs the answer text tag; The text is input into the first conversion module, and the first conversion module outputs a first question matrix; the first question matrix output by the first conversion module is input into the first single-layer fully connected network, and the first single-layer fully connected network outputs a KV matrix; the learnable Q matrix creation module creates a learnable Q matrix; the KV matrix output by the first single-layer fully connected network is input into the second conversion module, and the learnable Q matrix output by the learnable Q matrix creation module is input into the second conversion module, and the second conversion module outputs attribute features; the image embedding features output by the projector, the instruction text embedding features output by the first large language model embedding module, and the attribute features output by the second conversion module are input into the splicing module, and the splicing module outputs the first feature; the first feature output by the splicing module is input into the first large language model encoding module, and the first large language model encoding module outputs the predicted text embedding features; the predicted text embedding features output by the first large language model encoding module are input into the first large language model word segmenter, and the predicted text is obtained by decoding by the first large language model word segmenter; the second question attribute text in the predicted text output by the first large language model word segmenter is input into the first conversion module, and the first conversion module outputs the second question matrix.
6. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 5 is characterized in that: The specific method of inputting the first question attribute text of the answer text in the training set into the first conversion module and the first conversion module outputting the first question matrix includes: the severity label adopts a one-hot encoding scheme, the quantization value of the label "severe" is 100, the quantization value of the label "general" is 010, and the quantization value of the label "mild" is 001; the frequency label adopts a one-hot encoding scheme, the quantization value of the label "high incidence" is 100, the quantization value of the label "common" is 010, and the quantization value of the label "rare" is 001.
7. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 5 is characterized in that: The initial values of all elements of the first problem matrix are all 0; the first problem matrix has four dimensions, the first dimension corresponds to different samples, the second dimension corresponds to different violation scene category classification labels, the third dimension is the severity label quantization value, and the fourth dimension is the frequency label quantization value.
8. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 1 is characterized in that: The training of the urban governance multimodal large model in step S2 using the training set obtained in step S1 includes first-stage training and second-stage training.
9. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 8 is characterized in that: The loss function used in the first stage of training is the first loss function: in, Represents the total loss during the training phase; represents the text space cross entropy loss; Represents the attribute classification cross entropy loss; α represents the adjustment coefficient of the attribute classification cross entropy loss influence; Among them, the text space cross entropy loss The calculation formula is: ∩ represents the probability distribution cross entropy loss calculation function; Represents the predicted text embedding features output by the first language model encoding module; Represents the answer text embedding features output by the first language model embedding module; Among them, attribute classification cross entropy loss The calculation formula is: in, represents the second problem matrix; Denotes the first problem matrix.
10. The method for constructing a large multimodal model of urban governance based on problem attributes according to claim 8 is characterized in that: The second stage training uses the first loss function as the loss function, and also uses the problem matrix similarity measure to evaluate the urban governance multimodal large model. The parameters of the urban governance multimodal large model with the largest problem matrix similarity measure are used as the parameters of the trained urban governance multimodal large model. The formula for the problem matrix similarity measure is: Among them, Sim represents the similarity measure of the problem matrix; Min represents the minimum value; Max represents the maximum value; Ham represents the Hamming distance calculation function; Mat Train Represents the components of different matrices composed of the first dimension in the first problem matrix.