Accumulated water depth extraction method based on direct preference optimization multi-modal large language model

Abstract

The invention provides an accumulated water depth extraction method based on a direct preference optimization multi-modal large language model. The method comprises the following steps: obtaining social media content to be identified; the to-be-recognized social media content is input into a pre-trained water depth recognition model, a water depth recognition result output by the water depth recognition model is obtained, and the water depth recognition model is a multi-modal large language model obtained through training based on a token-level direct preference optimization strategy. According to the ponding water depth extraction method based on the direct preference optimization multi-modal large language model, the multi-modal large language model is trained through the token-level direct preference optimization strategy to obtain the water depth recognition model; therefore, the water depth recognition model can more accurately recognize the corresponding water depth information according to the input to-be-recognized social media content in the target time period of the target area.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence application technology, and in particular to a method for extracting ponding depth based on a direct preference optimization multimodal large language model. Background Art

[0002] Waterlogging or flooding caused by heavy rainfall is one of the most common disasters. Waterlogging caused by floods not only hinders traffic and causes blockages, but also damages infrastructure such as streets and buildings. Collecting and identifying water depth information during floods can directly reflect the severity of floods, thereby facilitating the implementation of emergency measures. Therefore, it is very important to identify water depth during floods.

[0003] Limited by meteorological factors and observation conditions, traditional sources of waterlogging or flood disaster data are increasingly unable to meet the needs of increasingly frequent flood disasters. When real-time monitoring of flood events is carried out through airborne or remote sensing satellite images, severe weather conditions will limit the visibility of the images, especially during periods of heavy rainfall. At the same time, the satellite's re-entry time also limits the timeliness of the data. High-resolution optical remote sensing satellites usually need to obtain high-resolution images several days after the incident. Finally, underground areas such as garages and subways cannot be obtained through remote sensing images. There are also certain problems with other methods. For example, due to the limited distribution of water level observation sensors, large-scale monitoring cannot be achieved, and flood simulation models based on surface water and underground pipe networks cannot achieve timely monitoring due to complex physical mechanisms and the need for a large amount of computing resources.

[0004] Compared with traditional data sources, crowd-sourced social media big data can collect real-time spatiotemporal information related to urban flooding topics. When extreme disasters or emergencies occur, people will spontaneously seek help and upload disaster information and transmit information through social media. By collecting social media big data, the development process of each stage of the disaster can be reflected, which is helpful for pre-disaster prevention, emergency response and post-disaster recovery. Since crowd-sourced social media big data is low in cost to collect, collecting reports from the public to reflect the waterlogged areas in the city is a feasible way to help improve the perception of urban flood depth information.

[0005] Therefore, how to accurately extract water depth information based on complex social media data remains an urgent problem to be solved. Summary of the invention

[0006] The present invention provides a method for extracting accumulated water depth based on a direct preference optimized multimodal large language model, so as to solve the defect in the prior art that it is difficult to accurately extract water depth information from complex social media data, and realize a method for extracting accumulated water depth based on social media data.

[0007] The present invention provides a method for extracting ponding depth based on a direct preference optimization multimodal large language model, comprising:

[0008] Obtaining social media content to be identified;

[0009] The social media content to be identified is input into a pre-trained depth recognition model to obtain a depth recognition result output by the depth recognition model, wherein the depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

[0010] According to a method for extracting ponding depth based on a direct preference optimized multimodal large language model provided by the present invention, before the step of inputting the social media content to be identified into a pre-trained water depth identification model, the method further includes:

[0011] A multimodal large language model is trained using a direct preference optimization strategy. During the training process, weights for different text modules are added based on the original score calculation of the direct preference optimization strategy.

[0012] According to a method for extracting ponding depth based on direct preference optimization of a multimodal large language model provided by the present invention, before the step of training the multimodal large language model using a direct preference optimization strategy, the method further includes:

[0013] Obtain multiple pieces of social media data related to water depth, and filter out the social media data containing geographic locations;

[0014] Social media data of different modalities are annotated to obtain a data set for training the water depth recognition model.

[0015] According to a method for extracting pond depth based on direct preference optimization of a multimodal large language model provided by the present invention, the step of labeling social media data of different modalities specifically includes:

[0016] For text-based data, the geographic location and water depth information in the text data are extracted and standardized according to the preset format;

[0017] For image modality data, a plurality of instructions for asking questions to the image data are predetermined, and the instructions and each piece of image data are sequentially input into the multimodal large language model, and a recognition result output by the multimodal large language model is obtained, and based on the recognition result, the erroneous part thereof is corrected to obtain a correct reply result, and the correct reply result is used as a positive sample, and the recognition result is used as a negative sample;

[0018] For data in video mode, a frame of video data is extracted at a preset time interval and annotated as image data.

[0019] According to a method for extracting ponding depth based on direct preference optimization of a multimodal large language model provided by the present invention, after the step of obtaining the water depth recognition result output by the water depth recognition model, the method further includes:

[0020] Predefine water depth levels and classify water depth descriptions, and obtain a water depth level model based on the defined water depth levels;

[0021] The water depth identification result is input into the water depth grade model to obtain the water depth grade result output by the water depth grade model.

[0022] According to a method for extracting ponding depth based on direct preference optimization of a multimodal large language model provided by the present invention, after the step of obtaining the water depth grade result output by the water depth grade model, the method further includes:

[0023] The water depth level results are visualized.

[0024] The present invention also provides a device for extracting water depth based on direct preference optimization of a multimodal large language model, comprising:

[0025] An acquisition module, used for acquiring social media content to be identified;

[0026] The recognition module is used to input the social media content to be recognized into a pre-trained depth recognition model to obtain a depth recognition result output by the depth recognition model, wherein the depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

[0027] According to the present invention, a device for extracting water depth based on direct preference optimization multimodal large language model also includes:

[0028] A classification module is used to predefine water depth levels and classify water depth descriptions, and obtain a water depth level model based on the defined water depth levels;

[0029] The water depth identification result is input into the water depth grade model to obtain the water depth grade result output by the water depth grade model.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for extracting accumulated water depth based on direct preference optimization of a multimodal large language model as described above is implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for extracting accumulated water depth based on direct preference optimization of a multimodal large language model as described in any one of the above is implemented.

[0032] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for extracting accumulated water depth based on direct preference optimization of a multimodal large language model.

[0033] The method for extracting accumulated water depth based on direct preference optimization of a multimodal large language model provided by the present invention trains the multimodal large language model through a token-level direct preference optimization strategy to obtain a water depth recognition model, so that the water depth recognition model can more accurately recognize the corresponding water depth information according to the input social media content to be recognized in the target area and target time period, and finally obtain the water depth recognition result of the target area in the target time period according to multiple social media content to be recognized in the target area and target time period. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is one of the flow diagrams of the method for extracting the water depth of ponding based on the direct preference optimization multimodal large language model provided by the present invention;

[0036] Figure 2 It is a schematic diagram mainly used to show the optimization process of the token-level direct preference optimization strategy in the water depth extraction method based on the direct preference optimization multimodal large language model provided by the present invention;

[0037] Figure 3 It is a schematic diagram mainly used to display a water depth recognition model in the water depth extraction method based on direct preference optimization multimodal large language model provided by the present invention;

[0038] Figure 4 It is a schematic diagram mainly used to display the water depth level in the water depth extraction method based on direct preference optimization multimodal large language model provided by the present invention;

[0039] Figure 5 This is the second flow chart of the method for extracting water depth based on direct preference optimization multimodal large language model provided by the present invention;

[0040] Figure 6 It is a structural schematic diagram of a water depth extraction device based on a direct preference optimization multimodal large language model provided by the present invention;

[0041] Figure 7It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Combine the following Figures 1 to 4 The present invention introduces a method for extracting water depth based on direct preference optimization of a multimodal large language model. Figure 1 As shown, including:

[0044] Step 101, obtaining social media content to be identified;

[0045] The social media content to be identified includes content in text, image and video modalities or a combination thereof.

[0046] Specifically, when it is necessary to identify the water depth of the target area at the target time, social media publishing content containing geographic location information and water depth information is obtained from different social media platforms as the social media content to be identified, so as to identify the water depth information of the corresponding area based on the social media content to be identified.

[0047] Optionally, the social media may be a social media platform that mainly publishes text, images or videos.

[0048] Optionally, text content directly containing geographic location information may be determined as social media content to be identified that contains geographic location information, and social media content with positioning or tags may also be determined as social media content to be identified that contains geographic location information.

[0049] For example, when the text of the social media content to be identified is "There is water accumulation on Street A, which is up to the calves", it is considered to contain the geographic location information "Street A"; when the social media content to be identified contains the positioning information "Community B", it is considered to contain the geographic location information "Community B"; when the social media content to be identified contains the tag "#News in District C", it is considered to contain the geographic location information "District C".

[0050] Optionally, when water depth information is reflected in text, images and / or videos of social media content, it is confirmed as social media content to be identified that contains water depth information.

[0051] Optionally, the image content in the social media content includes static images and dynamic images.

[0052] Optionally, the social media content to be identified may be one or more pieces of content.

[0053] Step 102: input the social media content to be identified into a pre-trained depth recognition model to obtain a depth recognition result output by the depth recognition model, wherein the depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

[0054] Currently, most methods for extracting depth information based on social media data tend to develop a single-modal model for a single data source (such as text or image). This leads to a waste of other modal data resources in social media data, and is also prone to bias towards a single modality, ignoring important information carried in other modalities.

[0055] Therefore, in order to more flexibly realize water depth recognition based on social media data and to maximize the use of social media data for fine-grained extraction of water depth information, a multimodal large language model is used as the basic training to obtain a water depth recognition model in this embodiment.

[0056] In order to make full use of data of various modes in social media data, the selected multimodal large language model needs to be able to process large language model data of three modes: text, image and video. In this implementation, MiniCPM-V (a terminal-side multimodal large language model) is used as the basic model for explanation.

[0057] MiniCPM-V consists of a visual encoder, a compression layer, and a large language model. It is pre-trained on a Chinese corpus and can better adapt to social media content in a Chinese environment. The visual encoder is responsible for encoding images and video frames, the compression layer is responsible for compressing the visual encoding tokens into a reasonable range, and the large language model is responsible for understanding the compressed visual encoding and text encoding and outputting the results.

[0058] Although the multimodal large language model provides a basis for unified processing and encoding of data of different modalities, and unified understanding and processing can be achieved by encoding data of multiple modalities and converting them into natural language, the current large language model generally has hallucination problems, insufficient understanding of the water depth in the image, and is prone to output erroneous recognition results. Therefore, in order to release the model's ability to understand flood disasters and water levels and improve the accuracy of the model, in this implementation, the selected multimodal large language model is trained through a token-level direct preference optimization strategy to obtain a water depth recognition model with more accurate recognition results.

[0059] Among them, the token-level direct preference optimization strategy is different from the common direct preference optimization strategy. Instead, it refines the optimization process to the token level. Specifically, in the process of producing responses, text is generated token by token, and the weight of the text fragment corresponding to each token is adjusted to achieve more fine-grained adjustment and optimization, and avoid overfitting of the model while fine-tuning the large language model.

[0060] On this basis, the social media content to be identified is input into the trained water depth recognition model to obtain the water depth recognition result output by the model.

[0061] For example, if the text in the social media content to be identified is "The water has flooded my house", and contains the location "Community D", and the image content is "The water has flooded over the shoulder of the road", then the water depth identification model will generate an identification result such as "The water depth in Community D has submerged the shoulder of the road".

[0062] It can be understood that by collecting multiple social media contents to be identified in the target area during the target period and obtaining the water depth identification results, the water depth information of the target area during the target period can be obtained according to the water depth identification results.

[0063] The present invention trains a multimodal large language model through a token-level direct preference optimization strategy to obtain a water depth recognition model, so that the water depth recognition model can more accurately identify the corresponding water depth information according to the input social media content to be identified in the target area and target time period, and finally obtain the water depth recognition result of the target area in the target time period according to multiple social media contents to be identified in the target area and target time period.

[0064] In the method for extracting water depth from ponded water based on direct preference optimization of a multimodal large language model of the present invention, before the step of inputting the social media content to be identified into a pre-trained water depth identification model, the method further includes:

[0065] A multimodal large language model is trained using a direct preference optimization strategy. During the training process, weights for different text modules are added based on the original score calculation of the direct preference optimization strategy.

[0066] In order to train the initial multimodal large language model to obtain a water depth recognition model, the token direct preference optimization strategy is adopted to fine-tune the initial multimodal large language model. The loss value between positive and negative samples is selectively calculated through the token-based direct preference optimization loss function calculation method. The method of directly optimizing human preferences is based on the initial multimodal large language model answer. By comparing the positions of the corrected texts, the texts at these corresponding positions are optimized and trained.

[0067] Specifically, direct preference optimization transforms the reinforcement learning objective of human feedback into an equivalent supervised learning objective in a simple way. The key is to transform the reward function r(x, y) into the model to be optimized π * (y|x) and the reference model π ref (y|x), so that the model to be optimized can be optimized directly in an appropriate way on the preferred or corrected data.

[0068] Among them, the reward model r(x,y) can be expressed as:

[0069]

[0070] In the formula, β is a constant and Z(x) is the partition function. On this basis, the model based on manual preference optimization can be expressed as:

[0071]

[0072] The reference model usually consists of a base model adjusted for the instructions that one wants to improve, and is fixed during the preferred optimization, with only the model to be optimized being updated.

[0073] Furthermore, if Figure 2 As shown in the figure, based on the direct preference optimization strategy, in order to conduct more refined feedback and optimization at the text block or token level, it is necessary to add the weight of each text module to the original scoring calculation method based on the overall ranking. The original scoring calculation method can be expressed as:

[0074]

[0075] In the formula, y i is the i-th token of the response y. In order to make the corrected text segment y c In order to make a higher contribution to the score of the entire text, the above formula is further modified into a fine-grained weighted aggregation of multiple text fragments:

[0076]

[0077] Where γ is the weight hyperparameter of the corrected segment, γ>1, the larger the value, the more contribution the corrected segment makes to the overall score, N=y u +γy c , N is the normalization factor used for normalization to prevent longer answers from getting higher scores, y u is the text snippet corresponding to the original answer.

[0078] In simple terms, during the training process, the incorrect answers of the multimodal large language model in the previous round of training are improved so that it can learn and train more, thereby improving the model's ability to recognize the incorrect answers, and finally training a water depth recognition model.

[0079] On this basis, in a feasible implementation, the parameters of the multimodal large language model are fine-tuned using a method based on LORA (Low-Rank Adaptation of Large Language Models).

[0080] The principle is to transform the weight matrix of the model parameters The low-rank decomposition is used to represent the parameter update ΔW into two smaller low-rank matrices A and B, thereby reducing the number of parameters during model training, namely:

[0081]

[0082] During training, freeze the parameter W0 and only train the parameters of A and B. The forward propagation process of a certain layer model can be expressed as:

[0083] h=W0x+ΔWx=W0x+BAx;

[0084] The fine-tuning of the LORA method model can be regarded as a process of incrementing the original parameters. The incremental parameters, namely ΔW, can be approximated by two low-rank matrices, thereby reducing the number of parameters and improving the efficiency of training.

[0085] Furthermore, in order to evaluate the training effect of the multimodal large language model, in a feasible implementation method, the LORA training parameters are first merged, and then the test set inference results of the multimodal large language model at different rounds of parameters are merged. Finally, the inference results of each sample at different rounds are sorted according to the quality and correctness of the answer results, and the best round is confirmed according to the average ranking and variance of all samples, and the optimal model is selected and determined as the final water depth recognition model.

[0086] After the water depth recognition model is obtained by the above method, Figure 3 As shown, the collected social media content to be identified is input into it, and the model processes the data of each modality separately and inputs it into the large language model to obtain the water depth identification result output by the large language model. The water depth identification result describes the water depth of the target area.

[0087] In the method for extracting water depth based on direct preference optimization of a multimodal large language model of the present invention, before the step of training the multimodal large language model using a direct preference optimization strategy, the method further includes:

[0088] Obtain multiple pieces of social media data related to water depth, and filter out the social media data containing geographic locations;

[0089] In order to train a large multimodal language model based on a token-level direct preference optimization strategy, a corresponding sample database needs to be constructed.

[0090] Urban water depth extraction first needs to obtain and locate the location information of the text in social media data, which is the basis for waterlogging monitoring and emergency rescue. However, the lack of geographic location in crowdsource data hinders further spatial analysis and research on waterlogging events. At present, only 1%-2% of social media data are labeled with geographic location, and the location cannot fully represent the place to be described in the text. Most of the geographic location information is implicit in the text data. At present, it is still difficult to extract and locate geographic information in crowdsource data. The text expression in social media has the characteristics of diversity, flexibility and colloquialism, which is very different from the expression of standard addresses. Therefore, further exploration is needed.

[0091] In this implementation, social media data from multiple historical flood disasters are first collected, and data entries containing water depth information and geographic locations are screened out.

[0092] In a specific implementation, social media data during a flood disaster period is collected from a predetermined social media platform, aligned and pre-processed, and duplicate text content is removed.

[0093] Optionally, first extract the vector embedding of all social media texts through the text vector model, and traverse each data to calculate the cosine distance between the current text and all previous texts. If the distance is less than a certain threshold, such as less than 0.8, it is considered that the text is repeated with the previous text, and the text is discarded to achieve text filtering.

[0094] On this basis, a keyword list was developed to delete the data containing the keyword in the data set. The keyword list represents the keyword content that is not related to floods and waterlogging but appears repeatedly in large quantities.

[0095] Optionally, the keyword list includes the names of online KOLs (Key Opinion Leaders) and celebrities, which can be determined based on actual data acquisition conditions.

[0096] At this point, we have obtained social media data that is roughly related to flood disasters. We then use named entity recognition tools to identify the locations contained in the text and filter out data without location entities or data without coordinate information in the metadata. It can be understood that named entity recognition tools can identify text data in social media data and obtain locations directly mentioned in the text data; metadata is used to obtain location data of social media data to determine locations based on location.

[0097] Furthermore, in order to screen out social media data related to flood disasters, some random samples were manually labeled to obtain positive and negative examples of data items related to flood disasters. The binary classification flood disaster recognition model trained with the BERT model was used to identify positive examples on the data set after the initial screening.

[0098] Social media data of different modalities are annotated to obtain a data set for training the water depth recognition model.

[0099] After completing the collection of social media data related to flood disasters, it is necessary to design targeted processing and annotation for social media data types of different modalities, that is, to determine the labels of the samples and obtain sample data sets that can be used to train multimodal large language models.

[0100] In the method for extracting water depth based on direct preference optimization of a multimodal large language model of the present invention, the step of labeling social media data of different modalities specifically includes:

[0101] For text-based data, the geographic location and water depth information in the text data are extracted and standardized according to the preset format;

[0102] In a feasible implementation, for text data, the processing and annotation method is to formulate two named entities, one is the geographical location and the other is the water depth information.

[0103] In addition, considering that the description methods in social media datasets vary, if an entity is discontinuous, the entity is connected from small to large through relationship links. For example, when there is a discontinuous expression such as "Hongshan District: Luoyu Road, Guangba Road" in the text, it contains three geographical entities and actually has two geographical locations. At this time, "Luoyu Road" is pointed to "Hongshan District" and "Guangba Road" is pointed to "Hongshan District".

[0104] Furthermore, considering that there may be water depth descriptions of multiple locations in the text, the water depth is associated with the location through relational links. Finally, a set of lists is generated based on the annotation results. Each element in the list consists of a geographical location and water depth information. If some information is missing, the list will be empty.

[0105] For example, [{“geographic location”: “Luoyu Road, Hongshan District”, “water depth information”: “5cm”}…{…}…]. When input into the model, a set of prompt instructions are used to enable the model to extract the information in a specific format, such as JSON.

[0106] The geographical location is the extracted location information, and the water depth information can be direct numerical information such as "5cm", or a description of the water depth, such as "the water is above the instep".

[0107] For image modality data, a plurality of instructions for asking questions to the image data are predetermined, and the instructions and each piece of image data are sequentially input into the multimodal large language model, and a recognition result output by the multimodal large language model is obtained, and based on the recognition result, the erroneous part thereof is corrected to obtain a correct reply result, and the correct reply result is used as a positive sample, and the recognition result is used as a negative sample;

[0108] In a feasible implementation, the image data is processed and annotated by first formulating a series of instructions for asking questions of the image, which are related to flood disasters and water depth issues, and then inputting the image and instructions into the model to obtain the corresponding water depth information recognition results.

[0109] Based on the recognition results, the incorrect text is corrected to obtain the correct response, so that the corrected text is used as a positive sample and the original text of the model's answer is used as a negative sample for training. It can be understood that the model here refers to the initial multimodal large language model.

[0110] For data in video mode, a frame of video data is extracted at a preset time interval and annotated as image data.

[0111] In a feasible implementation, the video data is processed and annotated in a manner that a frame of video data is extracted at intervals of a preset duration as image data, and the image data is processed and annotated in the same manner as the image data.

[0112] Optionally, the preset duration can be determined according to the total duration of the video, and is set to 1 second in this embodiment.

[0113] Furthermore, it can be understood that, when the image is a dynamic image, it can be labeled in the same manner as video data.

[0114] In summary, a multimodal water depth sample database is finally formed through manual annotation, which is used to train the multimodal large language model. Specifically, during the training process, the data in the sample database is randomly divided into a training set and a validation set in a ratio of 8:2.

[0115] In the method for extracting water depth of accumulated water based on direct preference optimization of a multimodal large language model of the present invention, after the step of obtaining the water depth recognition result output by the water depth recognition model, the method further includes:

[0116] Predefine water depth levels and classify water depth descriptions, and obtain a water depth level model based on the defined water depth levels;

[0117] It can be understood that since the water depth recognition model is a large language model, the water depth recognition result output by the water depth recognition model is a descriptive text of the input content. In order to more intuitively determine the water depth corresponding to the output description text, a water depth level model can be pre-built and trained after the large language model.

[0118] In a specific implementation, water depth levels are first defined and water depth descriptions are categorized.

[0119] Specifically, if Figure 4 As shown, according to the reference objects in the water depth sample library, including but not limited to people, cars, bicycles, motorcycles, trucks, buses, etc., they are divided into different levels according to the characteristics described. The original reference object is a person, and the instep, ankle, calf, knee, thigh, hip, waist, chest, neck, and eyes without water accumulation and the person are divided into 11 levels. Then the corresponding descriptions of other reference objects are summarized into this category, such as the middle of the car wheel corresponds to the calf level, and the motorcycle seat corresponds to the thigh level.

[0120] On this basis, a water depth level model is constructed. Optionally, a pre-trained language model based on BERT (Bidirectional Encoder Representations from Transformers) is used as the basic model, followed by a fully connected layer to construct a water depth classification quantization model.

[0121] In other words, the BERT-based deep learning model is used to quantify and classify the water depth description, and finally the water depth recognition results are automatically graded and determined as the corresponding water depth level.

[0122] Furthermore, the data set required for training the water depth grade model is generated and randomly divided into training set and validation set in a ratio of 8:2 according to the pre-defined water depth grades. The data set is used to train and evaluate the water depth grade model, and the round result with the best performance on the validation set is selected as the water depth grade model finally used.

[0123] The water depth identification result is input into the water depth grade model to obtain the water depth grade result output by the water depth grade model.

[0124] After the water depth grade model is trained, it is connected to the water depth recognition model to directly output a predefined water depth grade based on the water depth recognition result generated by the water depth recognition model, thereby obtaining a more intuitive water depth recognition result.

[0125] In the method for extracting water depth based on direct preference optimization of a multimodal large language model of the present invention, after the step of obtaining the water depth grade result output by the water depth grade model, the method further includes:

[0126] The water depth level results are visualized.

[0127] Since the method for extracting accumulated water depth provided by the present invention is mainly used to extract the accumulated water depth caused by urban waterlogging, in order to more intuitively reflect the water depth conditions in various areas of the city, the obtained water depth level results can be visualized.

[0128] In a specific implementation, geocoding API (Application Programming Interface) is first used to geolocate social media data, and positioning results of different scales are obtained through the returned results.

[0129] Secondly, the social media content during the flood disaster was processed to extract and quantify its water depth information. If a piece of social media data involves the water depth of multiple instances at the same location, aggregation is performed through the truncated average method. For example, at the street scale, some data only describes the water depth of the city. These data will be ignored, and only the description of the street and more detailed geographic location will be considered, and the truncated average will be performed from them. The method of truncated averaging is to truncate the results of multiple instances according to the proportion and then take the average value, which can be expressed as:

[0130]

[0131] In the formula, X is the sorted water depth value, n is the number of instances, a is the truncation ratio, g is the integer part of n*a, and r is the fractional part of n*a, which can reduce the impact of extreme values ​​on the final result.

[0132] On this basis, the spatiotemporal distribution of water depth information at different scales is drawn according to the water depth grade results and geographical location information, and the visualization of the water depth grade results is completed. Figure 5 shown.

[0133] The following is a description of the accumulated water depth extraction device based on direct preference optimization of the multimodal large language model provided by the present invention. The accumulated water depth extraction device based on direct preference optimization of the multimodal large language model described below and the accumulated water depth extraction method based on direct preference optimization of the multimodal large language model described above can be referenced to each other.

[0134] like Figure 6 As shown, the water depth extraction device based on direct preference optimization multimodal large language model includes an acquisition module 601 and a recognition module 602:

[0135] An acquisition module 601 is used to acquire social media content to be identified;

[0136] The social media content to be identified includes content in text, image and video modalities or a combination thereof.

[0137] Specifically, when it is necessary to identify the water depth of the target area at the target time, social media publishing content containing geographic location information and water depth information is obtained from different social media platforms as the social media content to be identified, so as to identify the water depth information of the corresponding area based on the social media content to be identified.

[0138] Optionally, the social media may be a social media platform that mainly publishes text, images or videos.

[0139] Optionally, text content directly containing geographic location information may be determined as social media content to be identified that contains geographic location information, and social media content with positioning or tags may also be determined as social media content to be identified that contains geographic location information.

[0140] For example, when the text of the social media content to be identified is "There is water accumulation on Street A, which is up to the calves", it is considered to contain the geographic location information "Street A"; when the social media content to be identified contains the positioning information "Community B", it is considered to contain the geographic location information "Community B"; when the social media content to be identified contains the tag "#News in District C", it is considered to contain the geographic location information "District C".

[0141] Optionally, when water depth information is reflected in text, images and / or videos of social media content, it is confirmed as social media content to be identified that contains water depth information.

[0142] Optionally, the image content in the social media content includes static images and dynamic images.

[0143] Optionally, the social media content to be identified may be one or more pieces of content.

[0144] The identification module 602 is used to input the social media content to be identified into a pre-trained depth recognition model to obtain a depth recognition result output by the depth recognition model, wherein the depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

[0145] Currently, most methods for extracting depth information based on social media data tend to develop a single-modal model for a single data source (such as text or image). This leads to a waste of other modal data resources in social media data, and is also prone to bias towards a single modality, ignoring important information carried in other modalities.

[0146] Therefore, in order to more flexibly realize water depth recognition based on social media data and to maximize the use of social media data for fine-grained extraction of water depth information, a multimodal large language model is used as the basic training to obtain a water depth recognition model in this embodiment.

[0147] In order to make full use of data of various modes in social media data, the selected multimodal large language model needs to be able to process large language model data of three modes: text, image and video. In this implementation, MiniCPM-V (a terminal-side multimodal large language model) is used as the basic model for explanation.

[0148] MiniCPM-V consists of a visual encoder, a compression layer, and a large language model. It is pre-trained on a Chinese corpus and can better adapt to social media content in a Chinese environment. The visual encoder is responsible for encoding images and video frames, the compression layer is responsible for compressing the visual encoding tokens into a reasonable range, and the large language model is responsible for understanding the compressed visual encoding and text encoding and outputting the results.

[0149] Although the multimodal large language model provides a basis for unified processing and encoding of data of different modalities, and unified understanding and processing can be achieved by encoding data of multiple modalities and converting them into natural language, the current large language model generally has hallucination problems, insufficient understanding of the water depth in the image, and is prone to output erroneous recognition results. Therefore, in order to release the model's ability to understand flood disasters and water levels and improve the accuracy of the model, in this implementation, the selected multimodal large language model is trained through a token-level direct preference optimization strategy to obtain a water depth recognition model with more accurate recognition results.

[0150] Among them, the token-level direct preference optimization strategy is different from the common direct preference optimization strategy. Instead, it refines the optimization process to the token level. Specifically, in the process of producing responses, text is generated token by token, and the weight of the text fragment corresponding to each token is adjusted to achieve more fine-grained adjustment and optimization, and avoid overfitting of the model while fine-tuning the large language model.

[0151] On this basis, the social media content to be identified is input into the trained water depth recognition model to obtain the water depth recognition result output by the model.

[0152] For example, if the text in the social media content to be identified is "The water has flooded my house", and contains the location "Community D", and the image content is "The water has passed the shoulder of the road", then the water depth identification model will generate an identification result such as "The water depth in Community D has submerged the shoulder of the road".

[0153] It can be understood that by collecting multiple social media contents to be identified in the target area during the target period and obtaining the water depth identification results, the water depth information of the target area during the target period can be obtained according to the water depth identification results.

[0154] The present invention trains a multimodal large language model through a token-level direct preference optimization strategy to obtain a water depth recognition model, so that the water depth recognition model can more accurately identify the corresponding water depth information according to the input social media content to be identified in the target area and target time period, and finally obtain the water depth recognition result of the target area in the target time period according to multiple social media contents to be identified in the target area and target time period.

[0155] The water depth extraction device based on direct preference optimization multimodal large language model of the present invention also includes a classification module for pre-defining water depth levels and classifying water depth descriptions, and obtaining a water depth level model based on the defined water depth level training;

[0156] The water depth identification result is input into the water depth grade model to obtain the water depth grade result output by the water depth grade model.

[0157] It can be understood that since the water depth recognition model is a large language model, the water depth recognition result output by the water depth recognition model is a descriptive text of the input content. In order to more intuitively determine the water depth corresponding to the output description text, a water depth level model can be pre-built and trained after the large language model.

[0158] In a specific implementation, water depth levels are first defined and water depth descriptions are categorized.

[0159] Specifically, if Figure 4As shown, according to the reference objects in the water depth sample library, including but not limited to people, cars, bicycles, motorcycles, trucks, buses, etc., they are divided into different levels according to the characteristics described. The original reference object is a person, and the instep, ankle, calf, knee, thigh, hip, waist, chest, neck, and eyes without water accumulation and the person are divided into 11 levels. Then the corresponding descriptions of other reference objects are summarized into this category, such as the middle of the car wheel corresponds to the calf level, and the motorcycle seat corresponds to the thigh level.

[0160] On this basis, a water depth level model is constructed. Optionally, a pre-trained language model based on BERT (Bidirectional Encoder Representations from Transformers) is used as the basic model, followed by a fully connected layer to construct a water depth classification quantization model.

[0161] In other words, the BERT-based deep learning model is used to quantify and classify the water depth description, and finally the water depth recognition results are automatically graded and determined as the corresponding water depth level.

[0162] Furthermore, the data set required for training the water depth grade model is generated and randomly divided into training set and validation set in a ratio of 8:2 according to the pre-defined water depth grades. The data set is used to train and evaluate the water depth grade model, and the round result with the best performance on the validation set is selected as the water depth grade model finally used.

[0163] After the water depth grade model is trained, it is connected to the water depth recognition model to directly output a predefined water depth grade based on the water depth recognition result generated by the water depth recognition model, thereby obtaining a more intuitive water depth recognition result.

[0164] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a method for extracting the depth of water accumulation based on a direct preference optimization multimodal large language model, the method comprising: obtaining social media content to be identified; inputting the social media content to be identified into a pre-trained water depth recognition model, and obtaining a water depth recognition result output by the water depth recognition model, wherein the water depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

[0165] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for extracting accumulated water depth based on the direct preference optimization multimodal large language model provided by the above methods, and the method includes: obtaining social media content to be identified; inputting the social media content to be identified into a pre-trained water depth recognition model to obtain a water depth recognition result output by the water depth recognition model, wherein the water depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

[0167] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for extracting water depth from accumulated water based on a direct preference optimized multimodal large language model provided by the above-mentioned methods, the method comprising: obtaining social media content to be identified; inputting the social media content to be identified into a pre-trained water depth recognition model to obtain a water depth recognition result output by the water depth recognition model, wherein the water depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

[0168] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting water depth based on direct preference optimization of multimodal large language model, characterized in that: include: Obtaining social media content to be identified; The social media content to be identified is input into a pre-trained depth recognition model to obtain a depth recognition result output by the depth recognition model, wherein the depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

2. The method for extracting water depth based on direct preference optimization multimodal large language model according to claim 1 is characterized in that: Before the step of inputting the to-be-identified social media content into the pre-trained water depth identification model, the method further includes: A multimodal large language model is trained using a direct preference optimization strategy. During the training process, weights for different text modules are added based on the original score calculation of the direct preference optimization strategy.

3. The method for extracting water depth based on direct preference optimization multimodal large language model according to claim 1 is characterized in that: Before the step of using the direct preference optimization strategy to train the multimodal large language model, the method further includes: Obtain multiple pieces of social media data related to water depth, and filter out the social media data containing geographic locations; Social media data of different modalities are annotated to obtain a data set for training the water depth recognition model.

4. The method for extracting water depth based on direct preference optimization multimodal large language model according to claim 3 is characterized in that: The step of labeling social media data of different modalities specifically includes: For text-based data, the geographic location and water depth information in the text data are extracted and standardized according to the preset format; For image modality data, a plurality of instructions for asking questions to the image data are predetermined, and the instructions and each piece of image data are sequentially input into the multimodal large language model, and a recognition result output by the multimodal large language model is obtained, and based on the recognition result, the erroneous part thereof is corrected to obtain a correct reply result, and the correct reply result is used as a positive sample, and the recognition result is used as a negative sample; For data in video mode, a frame of video data is extracted at a preset time interval and annotated as image data.

5. The method for extracting ponding depth based on direct preference optimization multimodal large language model according to any one of claims 1 to 4, characterized in that: After the step of obtaining the water depth identification result output by the water depth identification model, the method further includes: Predefine water depth levels and classify water depth descriptions, and obtain a water depth level model based on the defined water depth levels; The water depth identification result is input into the water depth grade model to obtain the water depth grade result output by the water depth grade model.

6. The method for extracting ponding depth based on direct preference optimization multimodal large language model according to claim 5 is characterized in that: After the step of obtaining the water depth grade result output by the water depth grade model, the method further includes: The water depth level results are visualized.

7. A water depth extraction device based on direct preference optimization multimodal large language model, characterized in that: include: An acquisition module, used for acquiring social media content to be identified; The recognition module is used to input the social media content to be recognized into a pre-trained depth recognition model to obtain a depth recognition result output by the depth recognition model, wherein the depth recognition model is a multimodal large language model trained based on a token-level direct preference optimization strategy.

8. The device for extracting water depth based on direct preference optimization multimodal large language model according to claim 7 is characterized in that: Also includes: A classification module is used to predefine water depth levels and classify water depth descriptions, and obtain a water depth level model based on the defined water depth levels; The water depth identification result is input into the water depth grade model to obtain the water depth grade result output by the water depth grade model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for extracting accumulated water depth based on direct preference optimization of a multimodal large language model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for extracting accumulated water depth based on direct preference optimization of a multimodal large language model as described in any one of claims 1 to 6 is implemented.

Images