Urban governance large model detection method and system integrating multimodal data

Abstract

The present invention discloses a large-scale urban governance model detection method and system that integrates multimodal data. The method includes: obtaining image data of urban street scenes or urban blocks to obtain initial image data; obtaining the problem content that needs to be detected; constructing a text description set of urban management violations based on the problem content, and using the text thinking chain method to process the text description set; extracting text features from the simplified description set; performing feature matching and screening on the initial image data and text feature vectors; encoding image features on the initial image data combined with the matching results; calculating cosine similarity based on the text feature vector and the image feature vector, and converting it into a suspected problem score to determine whether there is an urban governance problem. By implementing the method of the present invention, the limitations of traditional urban management solutions in the face of rapidly changing urban environments can be resolved, and various urban management problems can be automatically identified and processed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a large-scale urban governance model detection method and system integrating multimodal data. Background Art

[0002] In the daily operation of modern cities, with the advancement of urbanization, urban management faces many complex and diverse challenges. Among them, problems such as occupying the road for business, illegal parking, illegal parking, and traffic congestion have become important factors affecting the smoothness of urban traffic and the quality of life of residents. These problems not only bring disorder to urban traffic, but also affect the travel efficiency and environmental quality of residents, and may even cause safety hazards and environmental pollution.

[0003] Traditionally, problems in urban management and governance usually rely on purpose-designed solutions. Most of these solutions rely on image recognition technology to identify and detect specific types of urban management problems by analyzing surveillance videos or other image data in the city. For example, identifying street occupation, illegal parking, or illegal advertising. These problems are solved by designing customized image recognition algorithms for each specific scenario, which usually requires a large amount of manually annotated data as training samples and requires continuous tuning of the model to ensure that it can maintain high accuracy and robustness in various environments.

[0004] However, as the needs of urban management continue to change and become increasingly complex, the traditional customized image recognition model development method has gradually exposed some significant limitations. Specifically, as the management problems faced by cities become more diverse and dynamic, solutions that are simply customized for specific problems are often difficult to quickly adapt to emerging problems. For example, cities may suddenly face new violations or emergencies, such as illegal occupation of public spaces and random parking of shared bicycles. These new problems may require the redesign of new image recognition models. If new models need to be redeveloped every time a new problem is encountered, it will often lead to slow responses and inability to respond to emergencies quickly and effectively. Each solution to a targeted problem requires a large amount of labeled data to train the image recognition model, and each new problem in urban management may require a new batch of data to be collected and labeled. This not only requires a lot of labor costs, but also increases the complexity of data storage, processing and management. In addition, as the number of urban management scenarios increases, the amount of data and computing resources required also surges, resulting in a significant increase in the cost of model training and optimization. This is a continuous and high economic burden for management departments. As urban management scenarios continue to expand, existing customized models often find it difficult to take into account all scenarios. The diversity and complexity of different problems lead to unstable performance of the model in different situations, and low recognition accuracy and high error rate may occur in some scenarios. With the increase in application scenarios, existing image recognition models may need to be constantly adjusted and optimized to adapt to new problems. This continuous optimization is not only time-consuming and labor-intensive, but may not achieve the expected results. Most traditional image recognition models are customized based on specific problem scenarios, which limits the flexibility and scalability of the models. When new types of problems or new urban management needs arise, the model's adaptability is relatively poor, and customized development often needs to be started from scratch. This approach is not only time-consuming, but also has poor versatility in different cities or different environments, and it is difficult to achieve cross-regional or cross-platform sharing and reuse.

[0005] Therefore, it is necessary to design a new method to address the limitations of traditional urban management solutions in the face of rapidly changing urban environments and to automatically identify and handle various urban management issues. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a large-scale urban governance model detection method and system that integrates multimodal data.

[0007] To achieve the above purpose, the present invention adopts the following technical solution: a large model detection method for urban governance integrating multimodal data, comprising:

[0008] Acquire image data of a city street scene or a city block to obtain initial image data;

[0009] Get the problem content that needs to be tested;

[0010] Constructing a text description set of urban management violations according to the problem content, and processing the text description set using a text thinking chain method to obtain a simplified description set;

[0011] Perform text feature extraction on the simplified description set to obtain a text feature vector;

[0012] Performing feature matching and screening on the initial image data and text feature vectors to obtain matching results;

[0013] Performing block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector;

[0014] The cosine similarity is calculated based on the text feature vector and the image feature vector, and converted into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

[0015] A further technical solution is: constructing a text description set of urban management violations, and using a text thinking chain method to process the text description set to obtain a simplified description set, including:

[0016] A text description set of urban management violations is constructed according to the problem content, wherein the text description set includes C= ,in The text representing the j-th problem;

[0017] The text thinking chain method is adopted to convert the logically complex description of the text description set into a long sentence composed of simple words and relations, so as to obtain a simplified description set.

[0018] A further technical solution is: extracting text features from the simplified description set to obtain a text feature vector, including:

[0019] Segmenting each simple description in the simplified description set, and converting the words obtained by the segmentation into word vectors of fixed dimensions through a text encoder;

[0020] The word vector is input into a multi-head self-attention mechanism to generate a text feature vector.

[0021] A further technical solution is: performing feature matching and screening on the initial image data and text feature vector to obtain a matching result includes:

[0022] Converting the text feature vector into a two-dimensional format to obtain a converted text feature vector;

[0023] Segmenting the initial image data to obtain a segmentation result;

[0024] Performing a convolution operation on each image block in the cutting result and the converted text feature vector to obtain an operation result;

[0025] A similarity score is calculated according to the operation result by using a Sigmoid function to obtain a matching result, wherein the matching result includes blocks corresponding to contents that may be consistent with the simplified description set and blocks corresponding to continuous related problem areas; a block with a high similarity score indicates that the corresponding area may contain contents that are consistent with the simplified description set, and adjacent blocks with high similarity scores belong to continuous related problem areas.

[0026] A further technical solution is: performing block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector, including:

[0027] Filtering the image blocks that meet the requirements according to the matching results, flattening the image blocks that meet the requirements into one-dimensional vectors, and mapping them to a high-dimensional space through a fully connected layer to obtain image word vectors;

[0028] The image word vector is deeply encoded using a multi-head self-attention mechanism to obtain an image feature vector.

[0029] A further technical solution is: the image blocks that meet the requirements are screened according to the matching results, and the image blocks that meet the requirements are flattened into a one-dimensional vector, and mapped to a high-dimensional space through a fully connected layer to obtain a picture word vector, including:

[0030] Retaining blocks corresponding to the matching results from all image blocks to obtain image blocks that meet the requirements;

[0031] The image blocks that meet the requirements are flattened into one-dimensional vectors and mapped to a high-dimensional space through a fully connected layer to obtain image word vectors.

[0032] A further technical solution is: performing cosine similarity calculation based on the text feature vector and the image feature vector and converting the result into a suspected problem score to determine whether the initial image data contains an urban governance problem corresponding to the problem content, including:

[0033] Calculating the cosine similarity between the text feature vector and the image feature vector, and converting it into a question suspicion score;

[0034] When the suspected problem score exceeds the set threshold, it is determined that the initial image data contains urban governance problems corresponding to the problem content.

[0035] A further technical solution is: calculating the cosine similarity between the text feature vector and the image feature vector and converting it into a question suspicion score includes:

[0036] The cosine similarity between the text feature vector and the image feature vector is calculated, and the cosine similarity is converted into a question suspicion score through a Softmax function.

[0037] A further technical solution is: performing cosine similarity calculation based on the text feature vector and the image feature vector and converting the result into a suspected problem score to determine whether the initial image data contains an urban governance problem corresponding to the problem content, including:

[0038] Splicing the text feature vector and the image feature vector to obtain a splicing result;

[0039] Inputting the splicing result into a plurality of expert models selected by a clustering mechanism in an expert model library to obtain a similarity score output by each expert model;

[0040] The similarity scores output by each expert model are weighted and summed to obtain the comprehensive similarity;

[0041] Converting the comprehensive similarity into a problem suspicion score;

[0042] When the suspected problem score exceeds the set threshold, it is determined that the initial image data contains urban governance problems corresponding to the problem content.

[0043] The present invention also provides a large-scale urban governance model detection system integrating multimodal data, including:

[0044] A picture data acquisition unit, used to acquire picture data of a city street scene or a city block to obtain initial picture data;

[0045] A question content acquisition unit, used to acquire question content that needs to be detected;

[0046] A description set determination unit, used to construct a text description set of urban management violations according to the problem content, and process the text description set using a text thinking chain method to obtain a simplified description set;

[0047] A text feature extraction unit, used for extracting text features from the simplified description set to obtain a text feature vector;

[0048] A matching unit, used for performing feature matching and screening on the initial image data and the text feature vector to obtain a matching result;

[0049] An image encoding unit, configured to perform block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector;

[0050] A problem determination unit is used to calculate cosine similarity based on the text feature vector and the image feature vector, and convert it into a problem suspicion score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention constructs a simplified text set of problems by acquiring urban street scene pictures and text descriptions of related problems; then, the descriptions are processed using a text thinking chain method to extract representative text feature vectors; then, the image data is matched with the text feature vectors to screen out relevant content, which is further encoded in combination with image features; finally, through cosine similarity calculation, it is converted into a suspected problem score, thereby realizing automatic identification and rapid processing of problems in urban governance, and improving management efficiency and response speed.

[0052] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0054] Figure 1 A schematic diagram of a flow chart of a large model detection method for urban governance integrating multimodal data provided by an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention Figure 1 ;

[0056] Figure 3 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention Figure 2 ;

[0057] Figure 4 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention Figure 3 ;

[0058] Figure 5 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention Figure 4 ;

[0059] Figure 6 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention Figure 5 ;

[0060] Figure 7 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention Figure 6 ;

[0061] Figure 8 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention Figure 7 ;

[0062] Fig. 9 A schematic diagram of a heat map corresponding to illegal parking of non-motor vehicles provided in an embodiment of the present invention;

[0063] Fig.10 A schematic diagram of a heat map corresponding to illegal parking of motor vehicles provided in an embodiment of the present invention;

[0064] Fig.11 A schematic diagram of the workflow of the expert model provided by an embodiment of the present invention;

[0065] Fig.12 A schematic block diagram of a large-scale urban governance model detection system integrating multimodal data provided by an embodiment of the present invention;

[0066] Fig.13 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0069] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0070] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0071] See also Figure 1 Figure 1 A schematic flow chart of a large model detection method for urban governance that integrates multimodal data provided in an embodiment of the present invention. The large model detection method for urban governance that integrates multimodal data is applied to a server. The server interacts with the terminal and the camera for data. In order to cope with the limitations of traditional urban management solutions, a simplified text set of the problem is first constructed by acquiring city street scene images and text descriptions of the problems to be detected. Then, the simplified description is converted into a text feature vector using text feature extraction technology. Then, the image data is matched and screened with the text feature vector, and the image features are extracted and encoded. Finally, a suspected problem score is generated through cosine similarity calculation, thereby automatically identifying and processing various types of urban management problems, thereby improving the adaptability and efficiency of the system.

[0072] Figure 1 Schematic diagram of the process of the urban governance large model detection method integrating multimodal data provided by the embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S170.

[0073] S110: Acquire image data of a city street scene or a city block to obtain initial image data.

[0074] In this embodiment, the initial image data refers to image data of a city street scene or a city block.

[0075] Specifically, the acquisition process relies on two main types of image acquisition devices, one is a portable mobile device (such as a smartphone, tablet, etc.), and the other is a fixed point camera (such as a traffic monitoring camera, security monitoring camera). Both devices are equipped with a camera function that can capture high-quality static images or video streams.

[0076] Mobile devices can be flexibly deployed in various corners of the city and carried by operators to different locations for shooting; while fixed-point cameras are installed at specific locations, such as intersections, both sides of streets or around public facilities, to provide continuous monitoring of a fixed area.

[0077] For mobile devices, the collection task can be started at any time as needed; for fixed cameras, it may be set to run 24 / 7 or activated according to a preset schedule. This depends on the specific application scenario and the required monitoring frequency. The collection process can be triggered manually (for example, the user presses the shutter button), automatically timed (automatically taking a photo at a certain interval), or based on event triggering (automatically taking a photo when a moving object such as a pedestrian or vehicle is detected).

[0078] To ensure that the captured images have sufficient resolution and clarity, the system will set corresponding parameter configurations, such as ISO sensitivity, shutter speed, focal length, etc., and may apply some image optimization techniques to improve the imaging effect.

[0079] Once the image is successfully captured, the next step is to transfer these raw image files to the cloud server via wireless networks (wired network, wireless network, etc.). In order to ensure the security and integrity of the data, encryption protocols are usually used during the transmission process. After arriving at the cloud, the raw image data will be immediately encoded and compressed to save storage space and speed up subsequent analysis and processing. At the same time, metadata tags such as geographic location information, acquisition timestamp, etc. may also be added at this stage for easy management and retrieval.

[0080] S120: Obtain the problem content that needs to be detected.

[0081] In this embodiment, the problem content to be detected may include any one of illegal parking, occupying the road for business, crowd gathering, traffic congestion, etc. The user can input the problem content to be detected according to the needs.

[0082] S130. Construct a text description set of urban management violations based on the problem content, and use a text thinking chain method to process the text description set to obtain a simplified description set.

[0083] In this embodiment, the text description set C of urban management violations refers to a set of detailed text descriptions of various urban management issues, and the simplified description set It refers to a set of more concise and clear descriptions processed through the text thinking chain method, which aims to optimize the performance of the image recognition system and enable it to more effectively serve the needs of urban governance.

[0084] In one embodiment, see Figure 2The above-mentioned step S130 may include steps S131~S132.

[0085] S131. Construct a text description set of urban management violations according to the problem content, wherein the text description set includes C= ,in The text representing the j-th problem.

[0086] In this embodiment, a text description set C is constructed based on the specific urban management problem content. This set C is a text representation of different types of violations (such as illegal parking, occupying the road for business, crowd gathering, traffic congestion, etc.), and each violation is represented by Represents, where j is an index from 1 to n, representing different violation types.

[0087] Specifically, this text description set C = is a set consisting of multiple elements, each element They are detailed text descriptions of a particular urban management violation. For example, is "illegal parking of non-motor vehicles on the sidewalk", These descriptions usually contain specific details about the violations, such as the location where they occurred, the objects involved, the nature of the behavior, etc., which constitute the basic data for subsequent processing.

[0088] S132. Using a text thinking chain method, the logically complex description of the text description set is converted into a long sentence consisting of simple words and relationships to obtain a simplified description set.

[0089] In this embodiment, a method called text thinking chain is used to process the text description set C constructed above. The main purpose of the text thinking chain method is to transform the logically complex problem description into a simpler and more intuitive form, that is, a long sentence composed of simple words and basic relationships. The key to this step is to identify and extract the core concepts and their relationships in the original description through semantic analysis and natural language processing technology, and then reorganize this information to form a new expression.

[0090] Simplified description set It refers to the new description set obtained after the text thinking chain method is processed. Each element t i (i=1, 2, 3,...) in represents a complex description A simplified version after conversion. For example, "illegal parking of non-motor vehicles on the sidewalk" may be converted to "cars or motor vehicles, located on the sidewalk".

[0091] Simplified descriptions are easier to match with image feature vectors because they use more basic language structures and reduce ambiguity in understanding. By breaking down complex problems into simple words and relationships, machine learning models can better capture the information contained in images, especially when faced with complex scenes. When encountering new urban management problems, existing simplified rules can be quickly applied without redesigning the entire process.

[0092] In addition, it should be noted that a predefined set of rules can be introduced to guide how to convert complex natural language descriptions into simple words and relations. These rules may be based on urban management regulations or previous experience in handling similar cases. Specially trained models are used to identify key entities in urban management violations, such as location, time, participants, etc., and standardize them. For example, "the sidewalk of XX Road" can be uniformly converted into the standard term "sidewalk". Considering that urban management violations often occur in a specific time and space context, the system needs to have a certain context understanding ability to capture and consider factors such as the time and geographical location of the event. For complex behavior descriptions, by analyzing sentence structure and semantic relationships, the logical chain is reconstructed so that the final long sentence retains the original information and is easy to understand and process. For example, "someone parks a vehicle in a prohibited parking area" can be converted to "someone, in a prohibited parking area, is illegally parking a vehicle." Combined with images or other forms of data input, the generation process of text thinking chains is optimized to ensure that the description is not only accurate but also consistent with the actual scene.

[0093] Specifically, rule-orientation is the basis for the generation of text thinking chains. It involves creating a set of detailed transformation rules that are designed to transform complex descriptions in natural language into simple, clear semantic expressions. This set of rules needs to take into account not only the specific requirements of urban regulations, but also the lessons learned from past cases. For example, when dealing with illegal parking issues, the rules may stipulate that "all descriptions of 'illegal parking' or 'random parking' should be unified as 'illegally parked vehicles'". In addition, expert systems can be introduced to dynamically adjust the rule set to adapt to the changing needs of urban management. Specifically, detailed rule definitions are formulated based on urban management regulations and lessons learned from past cases. These rules include the following:

[0094] Vocabulary replacement rules: used to replace specific words or phrases in natural language with standard terms. For example, descriptions such as "illegal parking" and "random parking" are uniformly converted to "illegally parked vehicles".

[0095] Sentence simplification rules: Simplify complex sentence structures to make the expression more direct and clear. For example, simplify "carried out activities in public green spaces without permission" to "carried out activities in public green spaces without permission".

[0096] Considering that urban management regulations may change over time and new types of cases may emerge, the rule set needs to be flexible and extensible:

[0097] Establish a user feedback mechanism, collect opinions and suggestions from front-line staff, and adjust unreasonable rules in a timely manner.

[0098] Use machine learning algorithms to automatically discover potential new rules and help the rule set evolve and improve continuously.

[0099] In this way, the rule set can not only improve the speed and accuracy of text conversion, but also ensure the consistency and standardization of the conversion results, thereby better serving urban governance.

[0100] Entity recognition is a key link to ensure the accuracy of the text thinking chain. Using machine learning algorithms or deep learning models, important entity information such as location, time, participants, etc. can be automatically extracted from the text and mapped to predefined standard terms. For example, "the sidewalk of XX Road" will be identified and standardized as "sidewalk", while "Tuesday, March 5, 2024 at 3 pm" will be simplified to "3 pm on a certain day". This standardization process helps to eliminate the understanding barriers caused by different expressions, and also facilitates subsequent data analysis and processing.

[0101] Contextual understanding enables the text thinking chain to better capture the specific time and space background of the event. By integrating technical means such as geographic information system (GIS) and time series database, the system can obtain the associated geographic location and time information while parsing the text. For example, when processing a traffic congestion report, the system can not only identify the fact that "a certain intersection has a serious congestion", but also associate it with the traffic flow data in the area during that period, thereby providing a more comprehensive understanding. This step is particularly important for formulating targeted solutions.

[0102] Logical reconstruction focuses on how to reasonably simplify complex sentence structures and semantic relationships into a form that is easy to understand and process. Specifically, it is to perform grammatical analysis on the original text, find out the connection between basic components such as subject, predicate and object, and then reorganize these components in a certain logical order. For example, the sentence "someone parked a car in a no-parking area" is reconstructed into "someone is illegally parking a car in a no-parking area", which not only maintains the original meaning but also improves readability. At the same time, this method also helps to reveal the actual behavior patterns hidden behind the text and provide support for decision-making.

[0103] Multimodal fusion refers to combining information from different sources, including but not limited to text, images, videos, etc., to form a richer and more complete description of the event. In urban governance, this usually means not only dealing with the text content of citizen complaints, but also analyzing the accompanying photos or video materials. By training a specialized multimodal model, it is possible to effectively integrate multiple forms of data, ensuring that the final generated text thinking chain is not only faithful to the original text, but also accurately reflects the actual situation. For example, when dealing with illegal parking cases, in addition to referring to text descriptions, photos taken on the spot can also be used to confirm the specific location of the violation and the characteristics of the vehicle, and then make a more accurate judgment.

[0104] S140: Perform text feature extraction on the simplified description set to obtain a text feature vector.

[0105] In this embodiment, a text feature vector refers to a piece of text content (such as a sentence or a paragraph) converted into a numerical vector of fixed length through a specific algorithm. The vector can effectively represent the semantic information and structural characteristics of the original text. Specifically, the text feature vector refers to the text features related to urban management violations.

[0106] In one embodiment, see Figure 3 , the above-mentioned step S140 may include steps S141~S142.

[0107] S141. Segment each simple description in the simplified description set, and convert the words obtained by the segmentation into word vectors of fixed dimensions through a text encoder.

[0108] In this embodiment, first, the input text description set is simplified into a series of simple descriptions. Then, each simple description is further segmented into independent words or symbols (i.e., tokens) to form a vocabulary list. For example, "illegal parking" is divided into two tokens, "illegal" and "parking".

[0109] Then, using the pre-trained text encoder network, these tokens are converted into numerical vectors of the same dimension, called word vectors. This process ensures that the semantic similarity between different tokens can be reflected in the geometric space, that is, words with similar semantics are also close in the vector space.

[0110] Each word vector represents the position of a token in this multidimensional space, and together they constitute a fixed-dimensional vector of the initial representation of the entire text. .

[0111] S142. Input the word vector into a multi-head self-attention mechanism to generate a text feature vector.

[0112] In this embodiment, these word vectors It is passed as input to a multi-layer structure, which contains multiple multi-head self-attention mechanisms connected end to end.

[0113] The multi-head self-attention mechanism allows the model to focus on different parts of the text at the same time and capture long-distance dependencies, thereby enhancing the understanding of the context. After this series of operations, the original word vector is converted into a deeper text feature vector ,.

[0114] These text feature vectors It not only retains the information of the original text, but also integrates the semantic associations in the context to form a richer and more abstract representation.

[0115] Text feature vectors can capture the complex relationships between words and the semantic meaning of the entire sentence, which is crucial for accurately understanding text. In tasks such as classification and clustering, text feature vectors help the system identify potential patterns in the data and improve prediction performance. Due to its powerful expressive power, text feature vectors can be widely used in different natural language processing scenarios, including but not limited to intelligent question and answer, dialogue systems, recommendation systems, etc.

[0116] S150: performing feature matching and screening on the initial image data and the text feature vector to obtain a matching result.

[0117] In this embodiment, the matching result refers to the information obtained after a series of processing steps about which parts of the initial image data may contain content that matches the simplified description set. Specifically:

[0118] For each image block, if its similarity score exceeds a certain predetermined threshold, it is considered that the content of this image block may contain elements that match the problem description. When multiple adjacent image blocks have high similarity scores, it indicates that these blocks constitute a continuous related problem area, which means that the image content in this area is likely to reflect the situation in the problem description as a whole. The matching results are usually presented in a visual way, such as highlighting suspected matching image blocks or areas, or directly returning the specific coordinates of these areas and other information, so that users can further analyze or take corresponding measures.

[0119] In one embodiment, see Figure 4 The above-mentioned step S150 may include steps S151~S153.

[0120] S151. Convert the format of the text feature vector into a two-dimensional format to obtain a converted text feature vector.

[0121] In this embodiment, the original one-dimensional text feature vector is converted into a two-dimensional format (for example, a 256-dimensional vector is converted into a 16x16 matrix), which is recorded as This step ensures that subsequent operations can be performed in a unified space, making the text feature vector Can be compared on the same dimensions as image data.

[0122] S152: Segment the initial image data to obtain a segmentation result.

[0123] In this embodiment, all input images are first adjusted to the same size (H, W) and the number of RGB channels is kept at 3, ensuring that all image data conforms to the three-dimensional tensor format of (3, H, W). This is done to ensure the consistency and efficiency of subsequent processing. Then, according to a preset size p (for example, 16x16), each image is divided into multiple fixed-size blocks, each of which has a size of (3, p, p). This step helps extract local features and allows the model to focus on different parts of the image.

[0124] S153, performing a convolution operation on each image block in the cutting result and the converted text feature vector to obtain an operation result.

[0125] In this embodiment, the initial image data is evenly divided into a number of 16x16 small blocks, each of which represents a local area of ​​the image. As a filter, it performs convolution operations with each image block in turn. Convolution is a mathematical operation that calculates the similarity or correlation between two sets of data by sliding a window. In this process, It's like a detector that finds which parts of the image contain Related element information.

[0126] S153. Calculate the similarity score through the Sigmoid function according to the operation result to obtain a matching result, wherein the matching result includes blocks corresponding to the content that may contain the simplified description set and blocks corresponding to the continuous related problem area; blocks with high similarity scores indicate that the corresponding area may contain the content that conforms to the simplified description set, and adjacent blocks with high similarity scores belong to the continuous related problem area.

[0127] In this embodiment, a Sigmoid activation function is applied to the result of the convolution operation to convert it into a value between 0 and 1, i.e., a similarity score. This score reflects the degree of match between the image block and the target feature (the problem represented by the simplified description set).

[0128] Based on a pre-set threshold, it is possible to determine which image blocks are more likely to contain content that matches the problem description. Blocks with higher scores mean that they are more likely to contain elements in the problem description; and blocks with consecutive high scores indicate that these areas may be connected and related problem areas.

[0129] By effectively combining image and text features, we can automatically filter out those image contents that are most likely to reflect the description of a specific problem, thereby helping decision makers to quickly locate and understand the problem.

[0130] S160 , performing block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector.

[0131] In this embodiment, the image feature vector refers to a numerical expression extracted from the original image that can represent the characteristics of the image content. Specifically, it is the final output after being processed by the image encoder, and contains the abstract features of each element in the image. These features not only reflect the visual attributes of the image itself, but may also include semantic information, such as object category, scene type, etc.

[0132] Image feature vectors are usually low-dimensional but rich in information, which makes them easy to store and retrieve quickly. This means that when you need to compare two images or find similar images in a large database, you can directly compare their feature vectors without reanalyzing the entire image.

[0133] In this specific application scenario, the image feature vector is mainly used to match the urban governance problem to be detected. That is, by comparing the similarity between the newly collected image feature vector and the known problem pattern, it can help identify and classify new urban governance challenges, such as traffic violations.

[0134] In one embodiment, see Figure 5 The above-mentioned step S160 may include steps S161~S162.

[0135] S161. Filter the image blocks that meet the requirements according to the matching results, flatten the image blocks that meet the requirements into one-dimensional vectors, and map them to a high-dimensional space through a fully connected layer to obtain a picture word vector.

[0136] In this embodiment, the matching results obtained in step S150, that is, the image blocks that are considered to contain problematic information, are used to filter out irrelevant image blocks. This operation improves the effectiveness of the subsequent encoding stage and reduces the waste of computing resources.

[0137] For each image block that passes the screening, it is expanded into a one-dimensional vector of dimension 3*p*p. If p=16, each image block is converted into a one-dimensional vector of dimension 768. Then, a fully connected layer is used to linearly project these one-dimensional vectors into a higher-dimensional space to obtain a D-dimensional image word vector. This transformation can capture more complex feature representations and enable features of different types to be compared on the same dimension.

[0138] In one embodiment, see Figure 6 , the above-mentioned step S161 may include steps S1611~S1612.

[0139] S1611. Retain blocks corresponding to the matching results from all image blocks to obtain image blocks that meet the requirements.

[0140] In one embodiment, the above-mentioned step S1611 may include steps S16111~S16112.

[0141] S16111, generating a heat map according to the matching result and all the image blocks;

[0142] S16112. Determine an image block that meets the requirements from the heat map, wherein the image block that meets the requirements includes a darker area in the heat map.

[0143] S1612: Flatten the image blocks that meet the requirements into one-dimensional vectors, and map them to a high-dimensional space through a fully connected layer to obtain image word vectors.

[0144] In this embodiment, starting from the original problem description (for example, "electric vehicles, bicycles, and tricycles are parked on the sidewalk and not in the marked area"), the descriptive words are expanded through the text thinking chain method, and these keywords are associated with the image content.

[0145] Apply a text content matching algorithm to each image patch to evaluate whether it matches the above description. This step can be done based on a pre-trained model that can understand and compare the relationship between text and visual data.

[0146] Based on the matching results, a heat map is generated on the original image, where the color depth indicates the correlation between the corresponding area and the text description. The darker areas in the heat map mean a higher match, which means that they are more likely to contain evidence of illegal parking of non-motor vehicles.

[0147] Darker areas are selected from the heat map, which represent image patches that are highly correlated with the text description. These image patches are considered "qualified" because they are likely to contain information about non-motor vehicles not parked in accordance with regulations.

[0148] Once eligible image patches have been identified, the next step is to prepare this data for further analysis:

[0149] For each selected image patch, expand it into a one-dimensional vector. Assuming the image patch size is (3, p, p), the dimension of the flattened vector will be 3*p*p.

[0150] Then, a fully connected layer is used to map these one-dimensional vectors to a higher-dimensional space to generate D-dimensional image-word vectors. This process not only improves the ability to express features, but also enables different types of features to be effectively compared and calculated on a unified dimension.

[0151] After the above step S161, the obtained thermal map is as follows Fig. 9 and Fig.10 As shown, the heat map corresponding to the illegal parking of non-motor vehicles and the heat map corresponding to the illegal parking of motor vehicles can be obtained, which can show which specific areas may have illegal parking problems.

[0152] S162. Use a multi-head self-attention mechanism to deeply encode the image word vector to obtain an image feature vector.

[0153] In this embodiment, finally, the multi-head self-attention mechanism is applied to the above-obtained image word vector The multi-head attention mechanism allows the model to focus on different parts of the image at different positions, thereby better understanding the image content. After this series of operations, the output is a D-dimensional image feature vector .

[0154] S170. Calculate cosine similarity based on the text feature vector and the image feature vector, and convert it into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

[0155] In one embodiment, see Figure 7 , the above-mentioned step S170 may include steps S171~S172.

[0156] S171. Calculate the cosine similarity between the text feature vector and the image feature vector, and convert it into a question suspicion score.

[0157] Specifically, the cosine similarity between the text feature vector and the image feature vector is calculated, and the cosine similarity is converted into a question suspicion score through a Softmax function.

[0158] When dealing with urban governance issues such as illegal parking of non-motor vehicles, the system needs to evaluate the degree of match between image data and text descriptions. This step is done by calculating the text feature vector quantity. With the image feature vector This is achieved by taking the cosine similarity between the two and converting it into a question suspicion score.

[0159] In order to convert the cosine similarity into a problem suspicion score that can be used for decision making, the Softmax function is used. The role of the Softmax function is to convert a set of numerical values ​​into a probability distribution so that the output value falls between 0 and 1, and the sum of all output values ​​is equal to 1. Here, the Softmax function is used to emphasize the results with high similarity and suppress the results with low similarity. Specifically, if only a single similarity value CosSim is considered, the problem suspicion score Score can be simplified to: ; This approach ensures that even when the cosine similarity is low, the corresponding score will be very close to 0; conversely, when the cosine similarity is high, the score will approach 1, which effectively reflects the degree of match between the two.

[0160] S172. When the suspected problem score exceeds a set threshold, it is determined that the initial image data contains an urban governance problem corresponding to the problem content.

[0161] In this embodiment, once the suspected problem score is obtained, the next step is to determine whether the score exceeds a preset threshold. This process includes: determining a reasonable threshold in advance based on experience or experiments. For example, if the threshold is set to 0.8, it means that only when the suspected problem score exceeds 0.8, it is considered that there is a significant urban governance problem.

[0162] Compare the calculated suspected problem score with the set threshold. If Score>Threshold, the system can determine that the initial image data does contain the urban governance problem corresponding to the problem content, such as illegal parking of non-motor vehicles. On the contrary, if Score≤Threshold, further investigation or verification may be required, or it can be directly considered that there is no obvious problem.

[0163] In this way, the system can accurately identify specific problems in urban governance based on the effective combination of text and image features, and support more intelligent and automated management decisions.

[0164] In one embodiment, see Figure 8 and Fig.11, the above-mentioned step S170 may include steps S171'~ S175'.

[0165] S171', concatenate the text feature vector and the image feature vector to obtain a concatenation result.

[0166] In this embodiment, the splicing result refers to the text feature vector quantity With the image feature vector Splicing to form a new comprehensive 2D feature map The feature map integrates the information of text and image, which is convenient for subsequent analysis.

[0167] S172', input the splicing result into a plurality of expert models selected by a clustering mechanism in an expert model library to obtain a similarity score output by each expert model.

[0168] In this embodiment, the system pre-builds a library of expert models based on a convolutional neural network (CNN) architecture. Each expert model is optimized for a specific type of scenario or problem, such as illegal parking detection or street stall identification. The expert model library can adapt to feature matching of problems in different scenarios, and different expert models are good at matching different types of problem features.

[0169] When a new text-image pair needs to be matched, not all expert models will participate in the calculation. Instead, a mechanism called "clustering route gate" is used to select the K expert models that are most suitable for the current task. Specifically, each expert model learns a feature center (F center ). For a given task, by comparing The distance between each Fcenter can select the closest K expert models for further analysis.

[0170] The selected K expert models will receive the concatenated feature maps As input, a text feature vector is calculated. With the image feature vector The score reflects the degree of match between the two.

[0171] S173', perform weighted summation on the similarity scores output by each expert model to obtain a comprehensive similarity.

[0172] In this embodiment, in order to obtain the final similarity score, the similarity scores output by the K expert models need to be weighted and summed according to a certain weight rule. The selection of weights can be determined based on factors such as the professionalism of each expert model or its performance in previous tasks.

[0173] In the above steps, a comprehensive similarity score is obtained by weighted summation.

[0174] S174', converting the comprehensive similarity into a question suspicion score;

[0175] S175'. When the suspected problem score exceeds the set threshold, it is determined that the initial image data contains an urban governance problem corresponding to the problem content.

[0176] In this embodiment, the above steps S174'-S175' are similar to the above steps S171-S172, and are not described again herein.

[0177] The method of this embodiment can automatically identify whether there is a predefined question type directly from the uploaded image, without the need for additional training or fine-tuning for each newly input question. The core of this method is to use pre-trained image encoders and text encoders to extract features and calculate the similarity between these feature vectors to determine whether the image content matches the set question description.

[0178] No additional training or fine-tuning is required. A significant advantage of this method is that it does not require separate model training or parameter adjustment for each specific social governance issue (such as illegal parking of non-motor vehicles, occupying the road for business, etc.). The user only needs to provide an appropriate problem description text, and the system can analyze the uploaded image based on this description to determine whether it contains relevant issues. This is because the image encoder and text encoder have been fully trained to effectively capture the key features in images and texts, making them widely applicable to a variety of different social governance scenarios.

[0179] In the process of feature extraction using the encoder, the image encoder is responsible for converting the uploaded image into a high-dimensional feature vector , and the text encoder is responsible for converting the given problem description text into the corresponding text feature vector The two encoders are able to extract valuable feature information for specific tasks because they are pre-trained using a large number of basic data sets during the training phase. These basic data sets not only include rich street scene image elements - such as vehicles, pedestrians, and traffic facilities - but also contain detailed text descriptions that detail the appearance, behavior, and logical relationships in the images.

[0180] During pre-training, the encoder learns how to match text descriptions with corresponding image features. This means that when given a description of a social governance issue, the encoder is able to identify which visual features are most relevant to this description. For example, if the question is about "littering", the encoder will pay special attention to items or behavior patterns that may appear in garbage accumulation scenes. This ability comes from training on a large amount of image-text pair data, ensuring that the encoder can accurately identify problems in different contexts.

[0181] Calculate similarity to detect problems Once the image feature vector is obtained and text feature vector , the next step is to calculate the cosine similarity between the two and evaluate the degree of match between them. This similarity value is further processed by the Softmax function to obtain a suspected problem score (Score) indicating the possibility of the problem in the image. If this score exceeds the predetermined threshold, it can be considered that the corresponding problem does exist in the image.

[0182] It can be seen that the method of this embodiment can intelligently detect social governance issues from uploaded pictures without additional training by combining powerful image and text encoders. The encoder is pre-trained with a large amount of basic data, and can effectively extract feature information from pictures and texts, and quickly locate problems by calculating the similarity between feature vectors. This method not only improves detection efficiency, but also simplifies the deployment and maintenance process of the system, providing strong technical support for urban governance.

[0183] For example, clearly state the problem to be detected in text form, such as "illegal parking of non-motor vehicles on the sidewalk". This step is the basis of the entire process, and it determines the direction and scope of subsequent analysis. Next, use the text thinking chain method to further break down the problem to be detected into more specific object descriptions. For "illegal parking of non-motor vehicles", it can be concretized into the following aspects:

[0184] Vehicle type: including but not limited to electric vehicles, bicycles, tricycles, etc.

[0185] Parking location: Specifically indicate whether these vehicles are parked on the sidewalk.

[0186] Parking regulations: Confirm whether the vehicle is parked in the designated marked area.

[0187] In this way, you can ensure that the problem description is detailed enough, which will help improve the accuracy of subsequent analysis.

[0188] Using the pre-trained text encoder, the above detailed text description is converted into a feature vector in a high-dimensional space This feature vector contains all the key information about non-motor vehicle illegal parking and is ready to be compared with the image data.

[0189] Upload or input the image data to be analyzed and use the image encoding tool to convert it into the corresponding data vector This process also relies on a well-trained image encoder that can effectively capture the visual features in the image.

[0190] Finally, by calculating the text feature vector With the image feature vector The cosine similarity between the two is used to evaluate the degree of match between the two. In order to understand this matching relationship more intuitively, visualization techniques can also be used, such as generating a heat map to highlight the areas in the picture that are highly correlated with the text description. If there are significant matching results within the range of the heat map, it can be considered that there is indeed a "non-motor vehicle illegal parking" situation in the picture.

[0191] Specifically, this embodiment first identifies non-motor vehicles (such as electric vehicles, bicycles, and tricycles) parked on the sidewalk and checks whether they are within the specified lined area. Then, through the fine matching of text content and image slices, combined with the assistance of heat maps, it is finally determined whether there is "illegal parking of non-motor vehicles" in the image. This method not only improves the accuracy and efficiency of detection, but also makes the whole process more transparent and easy to understand.

[0192] In summary, through a series of steps such as defining problems, refining descriptions, generating text features, processing image data, and calculating similarities, specific problems in social governance, such as "illegal parking of non-motor vehicles", can be detected efficiently and accurately, providing strong technical support for urban management.

[0193] The above-mentioned urban governance large-scale model detection method that integrates multimodal data constructs a simplified text set of problems by obtaining urban street scene pictures and text descriptions of related problems; then, the description is processed using the text thinking chain method to extract representative text feature vectors; then, the image data is matched with the text feature vector, relevant content is screened out, and further encoded in combination with image features; finally, through cosine similarity calculation, it is converted into a suspected problem score, thereby realizing automatic identification and rapid processing of problems in urban governance, and improving management efficiency and response speed.

[0194] Fig.12 is a schematic block diagram of a large-scale urban governance model detection system 300 integrating multimodal data provided by an embodiment of the present invention. Fig.12As shown, corresponding to the above-mentioned urban governance big model detection method integrating multimodal data, the present invention also provides a large-scale urban governance model detection system 300 integrating multimodal data. The large-scale urban governance model detection system 300 integrating multimodal data includes a unit for executing the above-mentioned urban governance big model detection method integrating multimodal data, and the system can be configured in a server. Specifically, please refer to Fig.12 The urban governance large model detection system 300 integrating multimodal data includes a picture data acquisition unit 301, a problem content acquisition unit 302, a description set determination unit 303, a text feature extraction unit 304, a matching unit 305, an image encoding unit 306 and a problem determination unit 307.

[0195] The image data acquisition unit 301 is used to acquire image data of urban street scenes or urban blocks to obtain initial image data; the problem content acquisition unit 302 is used to acquire the problem content that needs to be detected; the description set determination unit 303 is used to construct a text description set of urban management violations based on the problem content, and use the text thinking chain method to process the text description set to obtain a simplified description set; the text feature extraction unit 304 is used to extract text features from the simplified description set to obtain a text feature vector; the matching unit 305 is used to perform feature matching and screening on the initial image data and the text feature vector to obtain a matching result; the image encoding unit 306 is used to perform block filtering and image feature encoding on the initial image data combined with the matching result to obtain an image feature vector; the problem determination unit 307 is used to calculate the cosine similarity based on the text feature vector and the image feature vector, and convert it into a problem suspicion score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

[0196] In one embodiment, the description set determining unit 303 includes:

[0197] A set construction subunit is used to construct a text description set of urban management violations according to the problem content, wherein the text description set includes C= ,in A text representation of the j-th type of problem; a simplified subunit, used to convert the logically complex description of the text description set into a long sentence composed of simple words and relationships using a text thinking chain method to obtain a simplified description set.

[0198] In one embodiment, the text feature extraction unit 304 includes:

[0199] The word segmentation encoding subunit is used to segment each simple description in the simplified description set, and convert the words obtained by the segmentation into a word vector of fixed dimension through a text encoder; the vector conversion subunit is used to input the word vector into a multi-head self-attention mechanism to generate a text feature vector.

[0200] In one embodiment, the matching unit 305 includes:

[0201] A format conversion subunit is used to convert the format of the text feature vector into a two-dimensional format to obtain a converted text feature vector; a segmentation subunit is used to segment the initial image data to obtain a segmentation result; a convolution subunit is used to perform a convolution operation on each image block in the segmentation result and the converted text feature vector to obtain an operation result; a score matching subunit is used to calculate a similarity score through a Sigmoid function according to the operation result to obtain a matching result, wherein the matching result includes blocks corresponding to content that may contain content that is consistent with the simplified description set and blocks corresponding to continuous related problem areas; blocks with high similarity scores indicate that the corresponding areas may contain content that is consistent with the simplified description set, and adjacent blocks with high similarity scores belong to continuous related problem areas.

[0202] In one embodiment, the image encoding unit 306 includes:

[0203] A screening processing subunit is used to screen the image blocks that meet the requirements according to the matching results, flatten the image blocks that meet the requirements into a one-dimensional vector, and map them to a high-dimensional space through a fully connected layer to obtain a picture word vector; a deep encoding subunit is used to perform deep encoding on the picture word vector using a multi-head self-attention mechanism to obtain an image feature vector.

[0204] In one embodiment, the screening processing subunit includes:

[0205] A retention module is used to retain the blocks corresponding to the matching results from all image blocks to obtain image blocks that meet the requirements; a flattening mapping module is used to flatten the image blocks that meet the requirements into a one-dimensional vector, and map it to a high-dimensional space through a fully connected layer to obtain a picture word vector.

[0206] In one embodiment, the retention module includes:

[0207] A heat map generation submodule is used to generate a heat map based on the matching results and all the image blocks; an image block determination submodule is used to determine image blocks that meet the requirements from the heat map, wherein the image blocks that meet the requirements include the darker areas in the heat map.

[0208] In one embodiment, the problem determination unit 307 includes:

[0209] The calculation subunit is used to calculate the cosine similarity between the text feature vector and the image feature vector, and convert it into a problem suspicion score; the determination subunit is used to determine that the initial image data contains urban governance problems corresponding to the problem content when the problem suspicion score exceeds a set threshold.

[0210] In one embodiment, the calculation subunit is used to calculate the cosine similarity between the text feature vector and the image feature vector, and convert the cosine similarity into a question suspicion score through a Softmax function.

[0211] In one embodiment, the problem determination unit 307 includes:

[0212] A splicing subunit is used to splice the text feature vector and the image feature vector to obtain a splicing result; an input subunit is used to input the splicing result into a number of expert models screened by a clustering mechanism in an expert model library to obtain a similarity score output by each expert model; a weighted summation subunit is used to perform weighted summation on the similarity scores output by each expert model to obtain a comprehensive similarity; a conversion subunit is used to convert the comprehensive similarity into a suspected problem score; a determination subunit is used to determine that the initial image data contains an urban governance problem corresponding to the problem content when the suspected problem score exceeds a set threshold.

[0213] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned urban governance large model detection system 300 integrating multimodal data and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0214] The above-mentioned urban governance large model detection system 300 integrating multimodal data can be implemented in the form of a computer program. The computer program can be used in Fig.12 Runs on the computer device shown.

[0215] See also Fig.13 , Fig.13 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0216] See also Fig.13The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0217] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a large model detection method for urban governance that integrates multimodal data.

[0218] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0219] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a large-scale urban governance model detection method that integrates multimodal data.

[0220] The network interface 505 is used to communicate with other devices over the network. Fig.13 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0221] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0222] Acquire image data of urban street scenes or urban blocks to obtain initial image data; obtain the problem content that needs to be detected; construct a text description set of urban management violations based on the problem content, and use the text thinking chain method to process the text description set to obtain a simplified description set; perform text feature extraction on the simplified description set to obtain a text feature vector; perform feature matching and screening on the initial image data and the text feature vector to obtain a matching result; perform block filtering and image feature encoding on the initial image data combined with the matching result to obtain an image feature vector; perform cosine similarity calculation based on the text feature vector and the image feature vector, and convert it into a problem suspicion score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

[0223] In one embodiment, when the processor 502 implements the step of constructing a text description set of urban management violations and processing the text description set using a text thinking chain method to obtain a simplified description set, the processor 502 specifically implements the following steps:

[0224] A text description set of urban management violations is constructed according to the problem content, wherein the text description set includes C= ,in A textual expression representing the j-th problem; using a textual thinking chain method to convert the logically complex description of the textual description set into a long sentence consisting of simple words and relationships, so as to obtain a simplified description set.

[0225] In one embodiment, when the processor 502 implements the step of extracting text features from the simplified description set to obtain a text feature vector, the processor 502 specifically implements the following steps:

[0226] Each simple description in the simplified description set is segmented, and the words obtained by the segmentation are converted into word vectors of fixed dimensions through a text encoder; the word vectors are input into a multi-head self-attention mechanism to generate a text feature vector.

[0227] In one embodiment, when the processor 502 implements the step of performing feature matching and screening on the initial image data and the text feature vector to obtain a matching result, the processor 502 specifically implements the following steps:

[0228] The format of the text feature vector is converted into a two-dimensional format to obtain a converted text feature vector; the initial image data is segmented to obtain a cutting result; each image block in the cutting result is convolved with the converted text feature vector to obtain an operation result; a similarity score is calculated by a Sigmoid function according to the operation result to obtain a matching result, wherein the matching result includes blocks corresponding to contents that may be consistent with the simplified description set and blocks corresponding to continuous related problem areas; blocks with high similarity scores indicate that the corresponding areas may contain contents that are consistent with the simplified description set, and adjacent blocks with high similarity scores belong to continuous related problem areas.

[0229] In one embodiment, when the processor 502 implements the step of performing block filtering and image feature encoding on the initial image data in combination with the matching result to obtain the image feature vector, the processor 502 specifically implements the following steps:

[0230] According to the matching results, image blocks that meet the requirements are screened, and the image blocks that meet the requirements are flattened into one-dimensional vectors, mapped to a high-dimensional space through a fully connected layer to obtain image word vectors; the image word vectors are deeply encoded using a multi-head self-attention mechanism to obtain image feature vectors.

[0231] In one embodiment, when the processor 502 implements the step of screening the image blocks that meet the requirements according to the matching results, flattening the image blocks that meet the requirements into a one-dimensional vector, and mapping them to a high-dimensional space through a fully connected layer to obtain a picture word vector, the processor 502 specifically implements the following steps:

[0232] The blocks corresponding to the matching results are retained from all image blocks to obtain image blocks that meet the requirements; the image blocks that meet the requirements are flattened into one-dimensional vectors, and mapped to a high-dimensional space through a fully connected layer to obtain image word vectors.

[0233] In one embodiment, when the processor 502 implements the step of retaining the block corresponding to the matching result from all the image blocks to obtain the image block that meets the requirements, the processor 502 specifically implements the following steps:

[0234] A heat map is generated according to the matching results combined with all the image blocks; and image blocks that meet the requirements are determined from the heat map, wherein the image blocks that meet the requirements include areas with darker colors in the heat map.

[0235] In one embodiment, when the processor 502 implements the step of calculating the cosine similarity according to the text feature vector and the image feature vector and converting it into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data, the following steps are specifically implemented:

[0236] The cosine similarity between the text feature vector and the image feature vector is calculated and converted into a suspected problem score; when the suspected problem score exceeds a set threshold, it is determined that the initial image data contains an urban governance problem corresponding to the problematic content.

[0237] In one embodiment, when the processor 502 implements the step of calculating the cosine similarity between the text feature vector and the image feature vector and converting the cosine similarity into a suspected problem score, the processor 502 specifically implements the following steps:

[0238] The cosine similarity between the text feature vector and the image feature vector is calculated, and the cosine similarity is converted into a question suspicion score through a Softmax function.

[0239] In one embodiment, when the processor 502 implements the step of calculating the cosine similarity according to the text feature vector and the image feature vector and converting it into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data, the following steps are specifically implemented:

[0240] The text feature vector and the image feature vector are spliced ​​to obtain a splicing result; the splicing result is input into a plurality of expert models screened by a clustering mechanism in an expert model library to obtain a similarity score output by each expert model; a weighted sum is performed on the similarity scores output by each expert model to obtain a comprehensive similarity; the comprehensive similarity is converted into a suspected problem score; when the suspected problem score exceeds a set threshold, it is determined that the initial image data contains an urban governance problem corresponding to the problematic content.

[0241] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0242] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0243] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0244] Acquire image data of urban street scenes or urban blocks to obtain initial image data; obtain the problem content that needs to be detected; construct a text description set of urban management violations based on the problem content, and use the text thinking chain method to process the text description set to obtain a simplified description set; perform text feature extraction on the simplified description set to obtain a text feature vector; perform feature matching and screening on the initial image data and the text feature vector to obtain a matching result; perform block filtering and image feature encoding on the initial image data combined with the matching result to obtain an image feature vector; perform cosine similarity calculation based on the text feature vector and the image feature vector, and convert it into a problem suspicion score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

[0245] In one embodiment, when the processor executes the computer program to implement the steps of constructing a text description set of urban management violations and processing the text description set using a text thinking chain method to obtain a simplified description set, the processor specifically implements the following steps:

[0246] A text description set of urban management violations is constructed according to the problem content, wherein the text description set includes C= ,in A textual expression representing the j-th problem; using a textual thinking chain method to convert the logically complex description of the textual description set into a long sentence consisting of simple words and relationships, so as to obtain a simplified description set.

[0247] In one embodiment, when the processor executes the computer program to implement the step of extracting text features from the simplified description set to obtain a text feature vector, the processor specifically implements the following steps:

[0248] Each simple description in the simplified description set is segmented, and the words obtained by the segmentation are converted into word vectors of fixed dimensions through a text encoder; the word vectors are input into a multi-head self-attention mechanism to generate a text feature vector.

[0249] In one embodiment, when the processor executes the computer program to implement the step of performing feature matching and screening on the initial image data and the text feature vector to obtain a matching result, the processor specifically implements the following steps:

[0250] The format of the text feature vector is converted into a two-dimensional format to obtain a converted text feature vector; the initial image data is segmented to obtain a cutting result; each image block in the cutting result is convolved with the converted text feature vector to obtain an operation result; a similarity score is calculated by a Sigmoid function according to the operation result to obtain a matching result, wherein the matching result includes blocks corresponding to contents that may be consistent with the simplified description set and blocks corresponding to continuous related problem areas; blocks with high similarity scores indicate that the corresponding areas may contain contents that are consistent with the simplified description set, and adjacent blocks with high similarity scores belong to continuous related problem areas.

[0251] In one embodiment, when the processor executes the computer program to implement the step of performing block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector, the processor specifically implements the following steps:

[0252] According to the matching results, image blocks that meet the requirements are screened, and the image blocks that meet the requirements are flattened into one-dimensional vectors, mapped to a high-dimensional space through a fully connected layer to obtain image word vectors; the image word vectors are deeply encoded using a multi-head self-attention mechanism to obtain image feature vectors.

[0253] In one embodiment, when the processor executes the computer program to implement the step of screening the image blocks that meet the requirements according to the matching results, flattening the image blocks that meet the requirements into a one-dimensional vector, and mapping them to a high-dimensional space through a fully connected layer to obtain a picture word vector, the following steps are specifically implemented:

[0254] The blocks corresponding to the matching results are retained from all image blocks to obtain image blocks that meet the requirements; the image blocks that meet the requirements are flattened into one-dimensional vectors, and mapped to a high-dimensional space through a fully connected layer to obtain image word vectors.

[0255] In one embodiment, when the processor executes the computer program to implement the step of retaining the block corresponding to the matching result from all image blocks to obtain an image block that meets the requirements, the processor specifically implements the following steps:

[0256] A heat map is generated according to the matching results combined with all the image blocks; and image blocks that meet the requirements are determined from the heat map, wherein the image blocks that meet the requirements include areas with darker colors in the heat map.

[0257] In one embodiment, when the processor executes the computer program to implement the step of calculating the cosine similarity based on the text feature vector and the image feature vector and converting them into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data, the following steps are specifically implemented:

[0258] The cosine similarity between the text feature vector and the image feature vector is calculated and converted into a suspected problem score; when the suspected problem score exceeds a set threshold, it is determined that the initial image data contains an urban governance problem corresponding to the problematic content.

[0259] In one embodiment, when the processor executes the computer program to implement the step of calculating the cosine similarity between the text feature vector and the image feature vector and converting it into a suspected problem score, the processor specifically implements the following steps:

[0260] The cosine similarity between the text feature vector and the image feature vector is calculated, and the cosine similarity is converted into a question suspicion score through a Softmax function.

[0261] In one embodiment, when the processor executes the computer program to implement the step of calculating the cosine similarity based on the text feature vector and the image feature vector and converting them into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data, the following steps are specifically implemented:

[0262] The text feature vector and the image feature vector are spliced ​​to obtain a splicing result; the splicing result is input into a plurality of expert models screened by a clustering mechanism in an expert model library to obtain a similarity score output by each expert model; a weighted sum is performed on the similarity scores output by each expert model to obtain a comprehensive similarity; the comprehensive similarity is converted into a suspected problem score; when the suspected problem score exceeds a set threshold, it is determined that the initial image data contains an urban governance problem corresponding to the problematic content.

[0263] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0264] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0265] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0266] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the system of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0267] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0268] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A large-scale urban governance model detection method integrating multimodal data, characterized in that: include: Acquire image data of a city street scene or a city block to obtain initial image data; Get the problem content that needs to be tested; Constructing a text description set of urban management violations according to the problem content, and processing the text description set using a text thinking chain method to obtain a simplified description set; Perform text feature extraction on the simplified description set to obtain a text feature vector; Performing feature matching and screening on the initial image data and text feature vectors to obtain matching results; Performing block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector; wherein the image block is a block obtained by segmenting the initial image data, and the image feature encoding is deep encoding of an image word vector formed by flattening and mapping the image blocks that meet the requirements in the matching result; The cosine similarity is calculated based on the text feature vector and the image feature vector, and converted into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

2. The urban governance large model detection method integrating multimodal data according to claim 1 is characterized in that: The said constructing a text description set of urban management violations, and processing the text description set using a text thinking chain method to obtain a simplified description set, includes: A text description set of urban management violations is constructed according to the problem content, wherein the text description set includes C= ,in The text representing the j-th problem; The text thinking chain method is adopted to convert the logically complex description of the text description set into a long sentence composed of simple words and relations, so as to obtain a simplified description set.

3. The urban governance large model detection method integrating multimodal data according to claim 1 is characterized in that: The text feature extraction is performed on the simplified description set to obtain a text feature vector, including: Segmenting each simple description in the simplified description set, and converting the words obtained by the segmentation into word vectors of fixed dimensions through a text encoder; The word vector is input into a multi-head self-attention mechanism to generate a text feature vector.

4. The urban governance large model detection method integrating multimodal data according to claim 1 is characterized in that: The performing feature matching and screening on the initial image data and the text feature vector to obtain a matching result includes: Converting the text feature vector into a two-dimensional format to obtain a converted text feature vector; Segmenting the initial image data to obtain a segmentation result; Performing a convolution operation on each image block in the cutting result and the converted text feature vector to obtain an operation result; A similarity score is calculated according to the operation result by using a Sigmoid function to obtain a matching result, wherein the matching result includes blocks corresponding to contents that may be consistent with the simplified description set and blocks corresponding to continuous related problem areas; a block with a high similarity score indicates that the corresponding area may contain contents that are consistent with the simplified description set, and adjacent blocks with high similarity scores belong to continuous related problem areas.

5. The urban governance large model detection method integrating multimodal data according to claim 1 is characterized in that: The performing block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector includes: Filtering the image blocks that meet the requirements according to the matching results, flattening the image blocks that meet the requirements into one-dimensional vectors, and mapping them to a high-dimensional space through a fully connected layer to obtain image word vectors; The image word vector is deeply encoded using a multi-head self-attention mechanism to obtain an image feature vector.

6. The urban governance large model detection method integrating multimodal data according to claim 5 is characterized in that: The step of selecting the image blocks that meet the requirements according to the matching results, flattening the image blocks that meet the requirements into a one-dimensional vector, and mapping them to a high-dimensional space through a fully connected layer to obtain a picture word vector includes: Retaining blocks corresponding to the matching results from all image blocks to obtain image blocks that meet the requirements; The image blocks that meet the requirements are flattened into one-dimensional vectors and mapped to a high-dimensional space through a fully connected layer to obtain image word vectors.

7. The urban governance large model detection method integrating multimodal data according to claim 1 is characterized in that: The cosine similarity calculation is performed based on the text feature vector and the image feature vector, and converted into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data, including: Calculating the cosine similarity between the text feature vector and the image feature vector, and converting it into a question suspicion score; When the suspected problem score exceeds the set threshold, it is determined that the initial image data contains urban governance problems corresponding to the problem content.

8. The urban governance large model detection method integrating multimodal data according to claim 7 is characterized in that: The calculating the cosine similarity between the text feature vector and the image feature vector and converting the cosine similarity into a problem suspicion score includes: The cosine similarity between the text feature vector and the image feature vector is calculated, and the cosine similarity is converted into a question suspicion score through a Softmax function.

9. The urban governance large model detection method integrating multimodal data according to claim 1 is characterized in that: The cosine similarity calculation is performed based on the text feature vector and the image feature vector, and converted into a suspected problem score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data, including: Splicing the text feature vector and the image feature vector to obtain a splicing result; Inputting the splicing result into a plurality of expert models selected by a clustering mechanism in an expert model library to obtain a similarity score output by each expert model; The similarity scores output by each expert model are weighted and summed to obtain the comprehensive similarity; Converting the comprehensive similarity into a problem suspicion score; When the suspected problem score exceeds the set threshold, it is determined that the initial image data contains urban governance problems corresponding to the problem content.

10. The urban governance large model detection system integrating multimodal data is characterized by: include: A picture data acquisition unit, used to acquire picture data of a city street scene or a city block to obtain initial picture data; A question content acquisition unit, used to acquire question content that needs to be detected; A description set determination unit, used to construct a text description set of urban management violations according to the problem content, and process the text description set using a text thinking chain method to obtain a simplified description set; A text feature extraction unit, used for extracting text features from the simplified description set to obtain a text feature vector; A matching unit, configured to perform feature matching and screening on the initial image data and the text feature vector to obtain a matching result; wherein the image block is a block obtained by segmenting the initial image data, and the image feature encoding is a deep encoding of the image word vector formed by flattening and mapping the image blocks that meet the requirements in the matching result; An image encoding unit, configured to perform block filtering and image feature encoding on the initial image data in combination with the matching result to obtain an image feature vector; A problem determination unit is used to calculate cosine similarity based on the text feature vector and the image feature vector, and convert it into a problem suspicion score to determine whether there is an urban governance problem corresponding to the problem content in the initial image data.

Images