Atmospheric visibility detection method and device based on multi-modal model

Abstract

The invention discloses an atmospheric visibility detection method and device based on a multi-modal model, and relates to the technical field of artificial intelligence, and the detection method comprises the steps: obtaining image data and meteorological data; generating a preset question based on the image data and the meteorological data; and inputting a preset question into the target multi-modal model, outputting reply information for the preset question, and determining the atmospheric visibility of the position where the target device is located based on the reply information. According to the invention, the technical problem of low accuracy of atmospheric visibility detection in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a method and device for detecting atmospheric visibility based on a multi-modal model. BACKGROUND

[0002] With the development of image processing and deep learning technology, the visibility estimation methods generally used in the related art mainly include dark channel prior theory, structural similarity, deep neural network, etc., which indirectly infer the visibility by analyzing image features (such as brightness, contrast, color saturation, etc.).

[0003] However, the current atmospheric visibility analysis method has the problems of single data source, poor model adaptability, insufficient analysis accuracy, and difficulty in coping with complex weather environments.

[0004] At present, there is no effective solution to the above problems. SUMMARY

[0005] The embodiments of the present application provide a method and device for detecting atmospheric visibility based on a multi-modal model, to at least solve the technical problem of low accuracy of atmospheric visibility detection in the related art.

[0006] According to an aspect of an embodiment of the present application, a method for detecting atmospheric visibility based on a multi-modal model is provided, comprising: obtaining image data and weather data, wherein the image data is obtained by a monitoring device deployed on a target device, and the weather data is obtained by a preset weather device deployed on the target device; generating a preset question based on the image data and the weather data; inputting the preset question into a target multi-modal model to output a reply information for the preset question, and determining the atmospheric visibility of a location where the target device is located based on the reply information, wherein the target multi-modal model is one of a first multi-modal model and a second multi-modal model, the network structure of the first multi-modal model at least includes a first preset encoder and a time series network structure, and the network structure of the second multi-modal model at least includes a second preset encoder and a feature extraction network.

[0007] Further, before inputting the preset question into the target multi-modal model and outputting the reply information for the preset question, the method further comprises: collecting first historical video data and first historical meteorological data of the plurality of devices within a preset period of time, and frame extracting the first historical video data to obtain a plurality of first historical image data; determining a sequence data set based on all the first historical meteorological data and all the first historical image data, wherein the first historical meteorological data at least includes temperature, humidity, air pressure, wind speed and atmospheric visibility values; constructing a first question and text set based on the sequence data set; training the first initial multi-modal model based on the first question and text set to obtain the first multi-modal model.

[0008] Further, before inputting the preset question into the target multi-modal model and outputting the reply information for the preset question, the method further comprises: collecting second historical image data and second historical meteorological data of the plurality of devices at a plurality of time points to obtain a data set; constructing a second question and text set based on the data set; training the second initial multi-modal model based on the second question and text set to obtain the second multi-modal model.

[0009] Further, the step of constructing the first question and text set based on the sequence data set comprises: for each first historical image data and each first historical meteorological data in the sequence data set, generating a first meteorological question text based on the first historical image data and the temperature, humidity, air pressure and wind speed; generating a first meteorological answer text based on the atmospheric visibility value, wherein the first meteorological answer text is a reply to the first meteorological question text; and constructing the first question and text set based on all the first meteorological question texts and all the first meteorological answer texts.

[0010] Further, the step of constructing the second question and text set based on the data set comprises: generating a second meteorological question text based on the temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the data set; generating a second meteorological answer text based on the atmospheric visibility value; and constructing the second question and text set based on all the second meteorological question texts and all the second meteorological answer texts.

[0011] Further, the step of training the first initial multi-modal model based on the first question pair text set to obtain the first multi-modal model comprises: constructing a first preset encoder, wherein the first preset encoder is used to extract features of each first historical image data to obtain first image features, and a plurality of fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded into a first preset position of corresponding first meteorological question text in the first question pair text set; constructing a time sequence network structure, wherein the time sequence network structure is used to extract features of each first historical meteorological data to obtain first target time sequence features, and the first target time sequence features are embedded into a second preset position in the first meteorological question text; constructing the first initial multi-modal model based on the first preset encoder, the time sequence network structure and the preset multi-modal model; constructing a first preset feature vector, and embedding the first preset feature vector into a third preset position of the first meteorological question text and embedding the first preset feature vector into a fourth preset position of the first meteorological question text to obtain a first target question pair text set; and training the first initial multi-modal model based on the first target question pair text set according to a preset training strategy to obtain the first multi-modal model.

[0012] Further, the step of training the second initial multi-modal model based on the second question pair text set to obtain the second multi-modal model comprises: constructing a second preset encoder, wherein the second preset encoder is used to extract features of each second historical image data to obtain second image features, and a plurality of fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into a first preset position of corresponding second meteorological question text in the second question pair text set; constructing a feature extraction network, wherein the feature extraction network is used to extract features of each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into a second preset position in the second meteorological question text; constructing the second initial multi-modal model based on the second preset encoder, the feature extraction network and the preset multi-modal model; constructing a second preset feature vector, and embedding the second preset feature vector into a fifth preset position of the second meteorological question text and embedding the second preset feature vector into a sixth preset position of the second meteorological question text to obtain a second target question pair text set; and training the second initial multi-modal model based on the second target question pair text set according to a preset training strategy to obtain the second multi-modal model.

[0013] According to another aspect of the embodiments of the present application, a device for detecting atmospheric visibility based on a multi-modal model is also provided, comprising: an acquisition unit configured to acquire image data and meteorological data, wherein the image data is obtained by a monitoring device deployed on a target device, and the meteorological data is obtained by a preset meteorological device deployed on the target device; a generation unit configured to generate a preset question based on the image data and the meteorological data; and a determination unit configured to input the preset question into a target multi-modal model, output reply information for the preset question, and determine atmospheric visibility at a location of the target device based on the reply information, wherein the target multi-modal model is one of a first multi-modal model and a second multi-modal model, the network structure of the first multi-modal model at least comprises a first preset encoder and a time series network structure, and the network structure of the second multi-modal model at least comprises a second preset encoder and a feature extraction network.

[0014] Further, the device for detecting atmospheric visibility based on a multi-modal model further comprises: a first acquisition module configured to acquire first historical video data and first historical meteorological data of a plurality of devices within a preset period of time and frame the first historical video data to obtain a plurality of first historical image data before the preset question is input into the target multi-modal model to output the reply information for the preset question; a first determination module configured to determine a sequence data set based on all the first historical meteorological data and all the first historical image data, wherein the first historical meteorological data at least includes temperature, humidity, air pressure, wind speed, and atmospheric visibility values; a first construction module configured to construct a first question-text pair set based on the sequence data set; and a first training module configured to train a first initial multi-modal model based on the first question-text pair set to obtain the first multi-modal model.

[0015] Further, the device for detecting atmospheric visibility based on a multi-modal model further comprises: a second acquisition module configured to acquire second historical image data and second historical meteorological data of a plurality of devices at a plurality of time points to obtain a data set before the preset question is input into the target multi-modal model to output the reply information for the preset question; a second construction module configured to construct a second question-text pair set based on the data set; and a second training module configured to train a second initial multi-modal model based on the second question-text pair set to obtain the second multi-modal model.

[0016] Further, the first constructing module comprises: a first generating submodule, configured to generate, for each first historical image data and each first historical meteorological data in the sequence data set, a first meteorological question text based on the first historical image data and temperature, humidity, air pressure and wind speed; a second generating submodule, configured to generate a first meteorological answer text based on the atmospheric visibility value, wherein the first meteorological answer text is a reply to the first meteorological question text; and a first constructing submodule, configured to construct a first question pair text set based on all the first meteorological question texts and all the first meteorological answer texts.

[0017] Further, the second constructing module comprises: a third generating submodule, configured to generate a second meteorological question text based on temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the data set; a fourth generating submodule, configured to generate a second meteorological answer text based on the atmospheric visibility value; and a second constructing submodule, configured to construct a second question pair text set based on all the second meteorological question texts and all the second meteorological answer texts.

[0018] Further, the first training module comprises: a third constructing submodule, configured to construct a first preset encoder, wherein the first preset encoder is used to extract features of each first historical image data to obtain a first image feature, and a plurality of fully connected layers are used to process the first image feature to obtain a first target image feature, and the first target image feature is embedded into a first preset position of a corresponding first meteorological question text in the first question pair text set; a fourth constructing submodule, configured to construct a time series network structure, wherein the time series network structure is used to extract features of each first historical meteorological data to obtain a first target time series feature, and the first target time series feature is embedded into a second preset position in the first meteorological question text; a fifth constructing submodule, configured to construct a first initial multi-modal model based on the first preset encoder, the time series network structure and a preset multi-modal model; a sixth constructing submodule, configured to construct a first preset feature vector, and embed the first preset feature vector into a third preset position of the first meteorological question text and a fourth preset position of the first meteorological question text to obtain a first target question pair text set; and a first training submodule, configured to train the first initial multi-modal model based on a preset training strategy to obtain a first multi-modal model by using the first target question pair text set.

[0019] Further, the second training module comprises: a seventh constructing submodule, configured to construct a second preset encoder, wherein the second preset encoder is used to extract features of each second historical image data to obtain second image features, and a plurality of fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into a first preset position of corresponding second meteorological question text in the second question pair text set; an eighth constructing submodule, configured to construct a feature extraction network, wherein the feature extraction network is used to extract features of each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into a second preset position in the second meteorological question text; a ninth constructing submodule, configured to construct a second initial multi-modal model based on the second preset encoder, the feature extraction network and a preset multi-modal model; a tenth constructing submodule, configured to construct a second preset feature vector, and embed the second preset feature vector into a fifth preset position of the second meteorological question text and embed the second preset feature vector into a sixth preset position of the second meteorological question text to obtain a second target question pair text set; and a second training submodule, configured to train the second initial multi-modal model based on a preset training strategy by using the second target question pair text set to obtain a second multi-modal model.

[0020] According to another aspect of the embodiments of the present application, a computer program product is also provided, comprising a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement any of the above-mentioned methods for detecting atmospheric visibility based on a multi-modal model.

[0021] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising one or more processors and a memory, the memory being configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any of the above-mentioned methods for detecting atmospheric visibility based on a multi-modal model.

[0022] In the present application, image data and meteorological data are acquired, a preset question is generated based on the image data and the meteorological data, the preset question is input into a target multi-modal model, reply information for the preset question is output, and atmospheric visibility of a location where a target device is located is determined based on the reply information, thereby solving the technical problem of low accuracy of atmospheric visibility detection in the related art.

[0023] In the present application, real-time image and weather data are obtained from the monitoring device on the target device and the preset weather station, and based on the image data and the weather data, a preset question is generated, which is input into a target multi-modal model. The target multi-modal model can be a first multi-modal model, which has a network structure including at least a first preset encoder for processing multi-time sequence image data and a time sequence network for integrating weather time sequence data. The target multi-modal model can also be a second multi-modal model, which has a network structure including at least a second preset encoder for processing single-time sequence image data and a feature extraction network for extracting weather features. Through analysis of the target multi-modal model, reply information about atmospheric visibility can be output, and based on the reply information, atmospheric visibility at the location of the target device can be determined, thereby improving the accuracy of atmospheric visibility prediction under complex weather conditions. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a multi-modal model-based atmospheric visibility detection method is shown;

[0026] Figure 2 is a flowchart of a multi-modal model-based atmospheric visibility detection method according to Embodiment 1 of the present application;

[0027] Figure 3 is a schematic diagram of an optional question-answer pair according to an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of an optional multi-modal model reasoning according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of an optional multi-modal model structure using multi-time sequence image and weather data according to an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of an optional multi-modal model structure using single-time sequence image and weather data according to an embodiment of the present application;

[0031] Figure 7 is a flowchart of an optional atmospheric visibility prediction according to an embodiment of the present application;

[0032] Figure 8 is a schematic diagram of an optional multi-modal model-based atmospheric visibility detection device according to an embodiment of the present application;

[0033] Figure 9 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the personnel in the art better understand the present application scheme, the technical scheme in the embodiment of the present application will be described clearly and completely below in combination with the drawings in the embodiment of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] It should be noted that the relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with relevant laws, regulations and standards in relevant regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, the system and the relevant users or agencies are provided with an interface, before obtaining the relevant information, the interface needs to send an acquisition request to the aforementioned user or agency, and after receiving the consent information feedback from the aforementioned user or agency, the relevant information is acquired.

[0037] In this invention, during the data processing stage, images and meteorological data such as temperature, humidity, air pressure, and wind speed are processed in a targeted manner to generate question-answer pairs to meet the input requirements of the multimodal large model. During the multimodal large model training stage, two types of multimodal large models are constructed: one uses multi-time-series images and meteorological data, which has better accuracy, and the other uses single-time-series images and meteorological data, which has slightly lower accuracy but faster data processing and training speed. During the multimodal large model inference stage, multimodal data such as images and meteorological data collected by equipment mounted on the tower are used as input to the multimodal large model to ensure that atmospheric visibility analysis results can be output quickly and accurately under complex and changeable meteorological conditions.

[0038] The present invention will now be described in detail with reference to various embodiments.

[0039] Example 1

[0040] According to an embodiment of this application, an embodiment of an atmospheric visibility detection method based on a multimodal model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a method for detecting atmospheric visibility based on a multimodal model is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions may also be included. In addition, it may include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera, wherein the network interface can be connected to wired and / or wireless networks. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0042] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the atmospheric visibility detection method based on a multimodal model in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned atmospheric visibility detection method based on a multimodal model. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0045] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0046] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for detecting atmospheric visibility based on a multimodal model is shown. Figure 2 This is a flowchart of the atmospheric visibility detection method based on a multimodal model according to Embodiment 1 of this application, as follows: Figure 2 As shown, the method includes the following steps:

[0047] In step S201, image data obtained by a monitoring device deployed on the target device and weather data obtained by a preset weather device deployed on the target device are acquired, and a preset question is generated based on the image data and the weather data.

[0048] In the embodiment of the present application, image data collected by a monitoring device deployed on a target device (such as a communication tower) and weather data collected by a preset weather device (such as a micro weather station) deployed on the target device are acquired.

[0049] For example, a communication tower is equipped with a monitoring camera and a micro weather station, the monitoring camera captures a video, and the micro weather station records the temperature of 20°C, the humidity of 60%, the air pressure of 1013 hPa, and the wind speed of 5 m / s at that time.

[0050] In step S202, a preset question is generated based on the image data and the weather data.

[0051] In the embodiment of the present application, according to the acquired image data and weather data, a question is constructed according to a preset template (such as a question-answer pair template), thereby generating a preset question (for example, according to the following image (temperature of 20°C, humidity of 60%, air pressure of 1013 hPa, and wind speed of 5 m / s) <xxx>, humidity is <xxx>, the air pressure is <xxx>, the wind speed is <xxx>), please analyze the atmospheric visibility of the current scene in meters.

[0052] It should be noted that, represents the representation form of the image data collected from the monitoring camera after feature extraction.

[0053] Step S203, input the preset question to the target multi-modal model, output the reply information for the preset question, and determine the atmospheric visibility of the location where the target device is based on the reply information, wherein the target multi-modal model is one of the first multi-modal model and the second multi-modal model, the network structure of the first multi-modal model at least includes: a first preset encoder and a time series network structure, and the network structure of the second multi-modal model at least includes: a second preset encoder and a feature extraction network.

[0054] Figure 3 is a schematic diagram of an optional question-answer pair according to an embodiment of the present application, as Figure 3 shown, the question can be: "Atmospheric visibility distance can be analyzed by image, at the same time, temperature, humidity, air pressure, wind speed, etc. are related to atmospheric visibility distance, now the image is , the temperature is xxx, the humidity is xxx, the air pressure is xxx, the wind speed is xxx, please analyze the atmospheric visibility distance of the current scene according to the image, temperature, humidity, air pressure, wind speed and other factors", the answer can be: "According to the image, temperature, humidity, air pressure, wind speed and other factors, the current atmospheric visibility distance is xxx meters".

[0055] In the embodiment of the present application, the target multi-modal model refers to the multi-modal model after training, which can be the first multi-modal model or the second multi-modal model. The first multi-modal model is trained based on time series images using the first preset encoder and the time series network structure, while the second multi-modal model is trained based on single frame images using the second preset encoder and the feature extraction network. By inputting the preset question to the target multi-modal model, the reply information for the preset question is output, and based on the reply information, the atmospheric visibility of the location where the target device is (such as the atmospheric visibility of the current scene is 1000 meters) can be determined.

[0056] Figure 4 is a schematic diagram of an optional multi-modal large model inference according to an embodiment of the present application, as Figure 4 As shown, the monitoring camera and the micro weather station (i.e., micro visibility measurement station) are arranged on the communication tower, and the monitoring camera and the micro weather station are used to obtain multiple frames of images and multiple time series of temperature, humidity, air pressure, wind speed and other meteorological data at any time. The meteorological factors (i.e., meteorological data) such as temperature, humidity, air pressure and wind speed are input into the LSTM (Long Short-Term Memory, Long Short-Term Memory Network) model for feature extraction to obtain meteorological features. At the same time, the multiple frames of images are input into the 3D encoder for feature extraction to obtain image features, and the image features are processed through the full connection layer. The meteorological features and the processed image features are embedded in the corresponding positions of the question respectively, and the learnable prompt words are added before and after the question, and then input into the LLM (Large Language Model, Large Language Model) to answer the atmospheric visibility at the location of the communication tower.

[0057] In summary, the collected images and meteorological data are converted into questions to input into the target multi-modal model. The model can be selected in the first multi-modal model or the second multi-modal model. The first multi-modal model can process multi-time series information and has better accuracy, while the second multi-modal model can process single-time series information and has a lighter model. By selecting and determining the target multi-modal model, the atmospheric visibility prediction value can be output, overcoming the limitations of single modal data, fully utilizing the complementary advantages of images and meteorological data, improving the accuracy and adaptability of visibility analysis, and thus solving the technical problem of low accuracy of atmospheric visibility detection in related technologies.

[0058] In order to accurately obtain the first multi-modal model, in the atmospheric visibility detection method based on the multi-modal model provided in Embodiment 1 of the present application, first historical video data and first historical meteorological data of a plurality of devices within a preset period are collected, and the first historical video data is frame extracted to obtain a plurality of first historical image data. Based on all the first historical meteorological data and all the first historical image data, a sequence data set is determined, wherein the first historical meteorological data at least includes temperature, humidity, air pressure, wind speed and atmospheric visibility value. Based on the sequence data set, a first question pair text set is constructed. Based on the first question pair text set, a first initial multi-modal model is trained to obtain a first multi-modal model.

[0059] Optionally, the first historical meteorological data at least includes temperature, humidity, air pressure, wind speed and atmospheric visibility value, and can also include less or more meteorological data, such as adding rainfall, PM2.5 and other meteorological data, or reducing one or more data such as temperature, humidity, air pressure and wind speed.

[0060] In the embodiment of the present application, the first historical video data and the first historical meteorological data of a plurality of devices in a preset period (such as a week) are collected, the first historical video data is frame extracted to obtain a plurality of first historical image data, the key frames are extracted from the video data to reduce the data processing amount, and sufficient information is retained for feature analysis.

[0061] In the embodiment of the present application, the first historical image data and the first historical meteorological data at the corresponding time are integrated to form a sequenced data set (i.e. a sequence data set), so that the model can learn the influence of the time sequence, wherein the first historical meteorological data at least includes temperature, humidity, air pressure, wind speed and atmospheric visibility value, and based on the sequence data set, a first question pair text set is constructed (for example, question: considering and environmental parameters (temperature is <xxx>, humidity is <xxx>, the air pressure is <xxx>, the wind speed is <xxx>), the current visibility is predicted. Answer: The current visibility is <vis>After that, the first initial multi-modal model is trained based on the first question pair text set to obtain the first multi-modal model.

[0062] In order to accurately obtain the second multi-modal model, in the method for detecting atmospheric visibility based on the multi-modal model provided in Embodiment 1 of the present application, second historical image data and second historical meteorological data of a plurality of devices at a plurality of moments are collected to obtain a data set; a second question pair text set is constructed based on the data set; and a second initial multi-modal model is trained based on the second question pair text set to obtain the second multi-modal model.

[0063] In the embodiment of the present application, the first multi-modal model uses multi-frame image data and corresponding multi-time temperature, humidity, air pressure and wind speed data for model training and inference. Since multi-time information is used, the effect is better, but it is also more time-consuming. Therefore, a question pair using single-frame image and corresponding single-time temperature, humidity, air pressure and wind speed data can be used to train the model, and by collecting second historical image data and second historical meteorological data of a plurality of devices at a plurality of moments (such as different time points in a day), a data set is obtained. Based on the data set, a second question pair text set can be constructed, and the second question pair text set is used as training data to train a second initial multi-modal model to obtain a second multi-modal model.

[0064] In order to accurately construct the first question pair text set, in the method for detecting atmospheric visibility based on the multi-modal model provided in Embodiment 1 of the present application, for each first historical image data and each first historical meteorological data in the sequence data set, a first meteorological question text is generated based on the first historical image data and temperature, humidity, air pressure and wind speed; a first meteorological answer text is generated based on the atmospheric visibility value, wherein the first meteorological answer text is a reply to the first meteorological question text; and a first question pair text set is constructed based on all first meteorological question texts and all first meteorological answer texts.

[0065] In the embodiment of the present application, the sequence data set includes first historical image data and corresponding first historical meteorological data continuously collected in a period of time. For each first historical image data and each first historical meteorological data in the sequence data set, a first meteorological question text can be generated based on the first historical image data and temperature, humidity, air pressure and wind speed, and a first meteorological answer text (i.e. a reply to the first meteorological question text) can be generated based on the atmospheric visibility value. A first question pair text set is constructed based on all first meteorological question texts and all first meteorological answer texts.

[0066] In order to accurately construct the second question pair text set, in the method for detecting atmospheric visibility based on the multi-modal model provided in Embodiment 1 of the present application, the second meteorological question text is generated based on the temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the data set; the second meteorological answer text is generated based on the atmospheric visibility value; and the second question pair text set is constructed based on all the second meteorological question texts and all the second meteorological answer texts.

[0067] In the embodiment of the present application, the data set refers to the sum of the collected second historical image data and second historical meteorological data, the second meteorological question text is generated based on the temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the data set, the second meteorological answer text is generated based on the atmospheric visibility value, and the second question pair text set is constructed based on all the second meteorological question texts and all the second meteorological answer texts.

[0068] In order to improve the accuracy of the first multi-modal model, in the method for detecting atmospheric visibility based on the multi-modal model provided in Embodiment 1 of the present application, the first preset encoder is constructed, wherein the first image feature is obtained by performing feature extraction on each first historical image data by using the first preset encoder, and the first target image feature is obtained by processing the first image feature by using a plurality of fully connected layers, and the first target image feature is embedded into a first preset position of the corresponding first meteorological question text in the first question pair text set; the time series network structure is constructed, wherein the first target time series feature is obtained by performing feature extraction on each first historical meteorological data by using the time series network structure, and the first target time series feature is embedded into a second preset position in the first meteorological question text; the first initial multi-modal model is constructed based on the first preset encoder, the time series network structure and the preset multi-modal model; the first preset feature vector is constructed, and the first preset feature vector is embedded into a third preset position of the first meteorological question text and a fourth preset position of the first meteorological question text to obtain the first target question pair text set; and the first initial multi-modal model is trained based on the first target question pair text set by using a preset training strategy to obtain the first multi-modal model.

[0069] In the embodiments of the present application, a first preset encoder (such as a 3D encoder based on a Transformer (a kind of neural network architecture) architecture with a self-attention mechanism for processing video frames and spatial pixels) is constructed. By using the first preset encoder to extract features from each first historical image data, first image features are obtained. A plurality of fully connected layers are used to process the first image features to obtain first target image features (such as token form), which ensure that their format and dimension are adapted to the input requirements of the model, and the first target image features can be embedded into a first preset position (such as, Position) of the corresponding first meteorological question text in the first question pair text set. For example, the first question text can be: Please analyze the following image features ([token1, token2, …, tokenN]) and environmental parameters to predict the visibility.

[0070] Then, a time series network structure is constructed. By using the time series network structure (such as an LSTM network, which is not limited here) to extract features from each first historical meteorological data, time series features of the meteorological data (i.e., first target time series features) are obtained. The first target time series features are embedded into a second preset position in the first meteorological question text (such as, <xxx>The first question text in the first meteorological question text may be: please analyze the following image features ([token1, token2, …, tokenN]) and environmental parameters ([seq_temp, seq_hum, …, seq_wind]) to predict the visibility. The first initial multi-modal model may be constructed based on the first preset encoder, the time series network structure, and the preset multi-modal model (such as QwenLlama, deepseek, Intern, and other large models).

[0071] In the embodiment of the application, the first preset feature vector (i.e., a learnable prompt word) is constructed, and the first preset feature vector is embedded at a third preset position (such as the beginning of the first meteorological question text) of the first meteorological question text and at a fourth preset position (such as the end of the first meteorological question text) of the first meteorological question text, to obtain a first target question pair text set. The first initial multi-modal model is trained based on the preset training strategy and the first target question pair text set, to obtain the first multi-modal model. For example, two groups of learnable tokens (i.e., the first preset feature vector, such as <start_prompt> and <end_prompt>) are added at the beginning and the end of the first meteorological question text, respectively, to form the first target question pair text set: "<start_prompt> please analyze the following image features ([token1, token2, …, tokenN]) and environmental parameters ([seq_temp, seq_hum, …, seq_wind]) to predict the visibility. <end_prompt>".

[0072] Then, the first initial multi-modal model is trained based on the preset training strategy (such as next-token-prediction, a means of supervised learning) and the first target question pair text set, to obtain the first multi-modal model.

[0073] Figure 5 According to an optional multi-modal large model structure for utilizing multi-time sequence images and meteorological data, the multi-modal large model structure includes an LLM (i.e., a preset multi-modal model), a 3D encoder (i.e., a first preset encoder), an LSTM (i.e., a time series network structure), and a Connector (i.e., a full connection layer). First, the 3D encoder is used to extract features of multiple frames of images extracted from a video. The 3D encoder is composed of a Transformer, and the extracted features are in the form of tokens. Then, the Connector composed of multiple layers of full connection layers is used to transform the features, so as to align the input of the LLM. Finally, the output tokens are embedded in the question in the question-answer pair. The position, while the temperature, humidity, air pressure, wind speed and other meteorological factors corresponding to the multi-frame image are normalized and input into the LSTM network, the multi-time sequence temperature, humidity, air pressure and wind speed data are extracted, and finally the corresponding features extracted are embedded in the question-answer pair <xxx>After the position, the extracted multi-frame image features and the temperature, humidity, air pressure, and wind speed features are integrated into the question, the learnable prompt words are added before and after the question, and the model is trained based on the answer (i.e., the first target question pair text set). During the training process, the LLM is also fine-tuned to obtain a multi-modal large model (i.e., the first multi-modal model).

[0074] To improve the accuracy of the second multi-modal model, in the method for detecting atmospheric visibility provided in Embodiment 1 of the present application, a second preset encoder is constructed, wherein the second preset encoder is used to extract features from each second historical image data to obtain second image features, and a plurality of fully connected layers are used to process the second image features to obtain second target image features, which are embedded into a first preset position of the corresponding second meteorological question text in the second question pair text set; a feature extraction network is constructed, wherein the feature extraction network is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into a second preset position in the second meteorological question text; based on the second preset encoder, the feature extraction network, and the preset multi-modal model, a second initial multi-modal model is constructed; a second preset feature vector is constructed and embedded into a fifth preset position of the second meteorological question text, and a sixth preset position of the second meteorological question text to obtain a second target question pair text set; based on a preset training strategy, the second initial multi-modal model is trained using the second target question pair text set to obtain a second multi-modal model.

[0075] In the embodiment of the present application, a second preset encoder (a predefined encoder structure, which can be based on a Transformer architecture, for processing static image data) is constructed. By using the second preset encoder to extract features from each second historical image data, second image features are obtained, and a plurality of fully connected layers are used to process the second image features to obtain second target image features, which are embedded into a first preset position of the corresponding second meteorological question text in the second question pair text set (e.g., position).

[0076] Then, a feature extraction network is constructed, which is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into a second preset position in the second meteorological question text (e.g., <xxx>The second initial multi-modal model is constructed based on the second preset encoder, the feature extraction network, and the preset multi-modal model.

[0077] In the embodiment of the present application, the second preset feature vector (i.e., a learnable prompt word, used to strengthen the model understanding ability and context reasoning ability) is constructed, and the second preset feature vector is embedded at the fifth preset position (i.e., the beginning of the second meteorological question text) of the second meteorological question text and at the sixth preset position (i.e., the end of the second meteorological question text) of the second meteorological question text, to obtain a second target question pair text set. Based on a preset training strategy, the second initial multi-modal model is trained using the second target question pair text set to obtain a second multi-modal model.

[0078] Figure 6 An optional multi-modal large model structure using single time sequence images and meteorological data according to the embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the multi-modal large model structure includes an LLM (i.e., a preset multi-modal model), an encoder (i.e., a second preset encoder), a Transformer, and a Connector (i.e., a full connection layer). First, the encoder composed of the Transformer is used to extract features of the image. After the extracted features are in the form of tokens, the Connector composed of multiple full connection layers is used to transform the features to align the input of the LLM. Finally, the output tokens are embedded in the position of the question-answer pair. At the same time, the temperature, humidity, air pressure, wind speed, and other meteorological factors corresponding to the image are normalized and input into the feature extraction network composed of the Transformer to extract features of the temperature, humidity, air pressure, and wind speed data. Finally, the extracted corresponding features are embedded in the <xxx>After that, the extracted image features are integrated with temperature, humidity, air pressure, and wind speed features into the question, and learnable prompt words are added before and after the question. The model is trained based on the answer (i.e., the second target question pair text set), and the LLM is also fine-tuned during the training process to obtain a multi-modal large model (i.e., the second multi-modal model).

[0079] Figure 7 is an optional flowchart of atmospheric visibility prediction according to an embodiment of the present application, as shown in Figure 7 Collect images through the camera mounted on the communication tower, and collect corresponding weather data such as temperature, humidity, air pressure, wind speed, and atmospheric visibility distance at the weather station near the communication tower. Use the collected images and weather data to construct question-answer pairs, obtain a question-answer pair dataset, and use the question-answer pair dataset composed of images and weather data as training data for a multi-modal large model. Train the multi-modal large model to obtain a target multi-modal model. Then, use the images and weather data collected by the communication tower mounting device (i.e., the camera and the micro weather station) to construct a preset question, which is used as input for the target multi-modal large model. Perform multi-modal model inference to obtain the predicted atmospheric visibility value.

[0080] The method for detecting atmospheric visibility based on a multi-modal model provided in the embodiments of the present application combines the construction of a multi-modal question pair text set and the training of a multi-modal large model to achieve highly accurate prediction and real-time analysis of atmospheric visibility. First, the deep features of images and weather data are extracted and converted into question description parts in the question pair text set. Then, target image features and time series weather features are embedded in the question text, and preset feature vectors are added before and after the question to construct a target question pair text set. After that, the target question pair text set is used to train an initial multi-modal model using a preset training strategy, and two multi-modal models capable of processing single time series and multi-time series data are obtained. Through effective integration of images and weather data and training and fine-tuning of the multi-modal large model, atmospheric visibility can be accurately estimated based on real-time images and weather data, improving the accuracy and scene adaptability of visibility prediction.

[0081] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0082] Embodiment 2

[0083] The embodiment of the present application further provides a device for detecting atmospheric visibility based on a multi-modal model. It should be noted that the device for detecting atmospheric visibility based on a multi-modal model can be used to execute the method for detecting atmospheric visibility based on a multi-modal model provided by the embodiment of the present application. The device for detecting atmospheric visibility based on a multi-modal model provided by the embodiment of the present application is introduced as follows.

[0084] According to the embodiment of the present application, a device for implementing the above-mentioned method for detecting atmospheric visibility based on a multi-modal model is further provided. Figure 8 FIG. 1 is a schematic diagram of an optional device for detecting atmospheric visibility based on a multi-modal model according to the embodiment of the present application. As shown in FIG. 1, the device for detecting atmospheric visibility based on a multi-modal model can include an acquisition unit 80, a generation unit 81 and a determination unit 82. Figure 8

[0085] The acquisition unit 80 is configured to acquire image data and meteorological data. The image data is obtained by a monitoring device deployed on a target device, and the meteorological data is obtained by a preset meteorological device deployed on the target device.

[0086] The generation unit 81 is configured to generate a preset question based on the image data and the meteorological data.

[0087] The determination unit 82 is configured to input the preset question into a target multi-modal model, output reply information for the preset question, and determine atmospheric visibility at a location of the target device based on the reply information. The target multi-modal model is one of a first multi-modal model and a second multi-modal model. The network structure of the first multi-modal model at least includes a first preset encoder and a time series network structure. The network structure of the second multi-modal model at least includes a second preset encoder and a feature extraction network.

[0088] The device for detecting atmospheric visibility based on a multi-modal model provided by the embodiment of the present application can acquire image data and meteorological data through the acquisition unit 80, generate a preset question based on the image data and the meteorological data through the generation unit 81, input the preset question into a target multi-modal model through the determination unit 82, output reply information for the preset question, and determine atmospheric visibility at a location of a target device based on the reply information.

[0089] ​Optionally, the device for detecting atmospheric visibility based on a multi-modal model further comprises: a first acquisition module, configured to acquire first historical video data and first historical meteorological data of a plurality of devices within a preset period of time before inputting a preset question into a target multi-modal model and outputting reply information for the preset question, and frame the first historical video data to obtain a plurality of first historical image data; a first determination module, configured to determine a sequence data set based on all the first historical meteorological data and all the first historical image data, wherein the first historical meteorological data at least includes temperature, humidity, air pressure, wind speed and atmospheric visibility value; a first construction module, configured to construct a first question-text pair set based on the sequence data set; and a first training module, configured to train a first initial multi-modal model based on the first question-text pair set to obtain the first multi-modal model.

[0090] Optionally, the device for detecting atmospheric visibility based on a multi-modal model further comprises: a second acquisition module, configured to acquire second historical image data and second historical meteorological data of a plurality of devices at a plurality of time points to obtain a data set before inputting a preset question into a target multi-modal model and outputting reply information for the preset question; a second construction module, configured to construct a second question-text pair set based on the data set; and a second training module, configured to train a second initial multi-modal model based on the second question-text pair set to obtain the second multi-modal model.

[0091] Optionally, the first construction module comprises: a first generation submodule, configured to generate a first meteorological question text based on the first historical image data and temperature, humidity, air pressure and wind speed for each first historical image data and each first historical meteorological data in the sequence data set; a second generation submodule, configured to generate a first meteorological answer text based on the atmospheric visibility value, wherein the first meteorological answer text is a reply to the first meteorological question text; and a first construction submodule, configured to construct the first question-text pair set based on all the first meteorological question texts and all the first meteorological answer texts.

[0092] Optionally, the second construction module comprises: a third generation submodule, configured to generate a second meteorological question text based on temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the data set; a fourth generation submodule, configured to generate a second meteorological answer text based on the atmospheric visibility value; and a second construction submodule, configured to construct the second question-text pair set based on all the second meteorological question texts and all the second meteorological answer texts.

[0093] Optionally, the first training module comprises: a third construction submodule for constructing a first preset encoder, wherein the first preset encoder is used to extract features from each first historical image data to obtain first image features, and a plurality of fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded into a first preset position of corresponding first meteorological question text in the first question pair text set; a fourth construction submodule for constructing a time series network structure, wherein the time series network structure is used to extract features from each first historical meteorological data to obtain first target time sequence features, and the first target time sequence features are embedded into a second preset position in the first meteorological question text; a fifth construction submodule for constructing a first initial multi-modal model based on the first preset encoder, the time series network structure and the preset multi-modal model; a sixth construction submodule for constructing a first preset feature vector, and embedding the first preset feature vector into a third preset position of the first meteorological question text, and embedding the first preset feature vector into a fourth preset position of the first meteorological question text to obtain a first target question pair text set; and a first training submodule for training the first initial multi-modal model based on the preset training strategy to obtain the first multi-modal model by using the first target question pair text set.

[0094] Optionally, the second training module comprises: a seventh construction submodule for constructing a second preset encoder, wherein the second preset encoder is used to extract features from each second historical image data to obtain second image features, and a plurality of fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into a first preset position of corresponding second meteorological question text in the second question pair text set; an eighth construction submodule for constructing a feature extraction network, wherein the feature extraction network is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into a second preset position in the second meteorological question text; a ninth construction submodule for constructing a second initial multi-modal model based on the second preset encoder, the feature extraction network and the preset multi-modal model; a tenth construction submodule for constructing a second preset feature vector, and embedding the second preset feature vector into a fifth preset position of the second meteorological question text, and embedding the second preset feature vector into a sixth preset position of the second meteorological question text to obtain a second target question pair text set; and a second training submodule for training the second initial multi-modal model based on the preset training strategy to obtain the second multi-modal model by using the second target question pair text set.

[0095] The above-mentioned atmospheric visibility detection device based on the multi-modal model can further include a processor and a memory, and the above-mentioned acquisition unit 80, generation unit 81, determination unit 82, etc. are stored in the memory as program units, and the above-mentioned program units stored in the memory are executed by the processor to realize the corresponding functions.

[0096] The above-mentioned processor includes a core, and the core calls the corresponding program unit in the memory. The core can be one or more, and the preset problem is input into the target multi-modal model by adjusting the core parameters, the reply information for the preset problem is output, and the atmospheric visibility of the location of the target device is determined based on the reply information.

[0097] The above-mentioned memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0098] It should be noted that the above-mentioned acquisition unit 80, generation unit 81, and determination unit 82 correspond to steps S201 to S203 in Embodiment 1, and the above-mentioned units have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned units can be hardware components or software components stored in the memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a, 102b, …, 102n), and the above-mentioned units can also be a part of the device and can run in the computer terminal 10 provided in Embodiment 1.

[0099] Embodiment 3

[0100] The embodiments of the present application can provide a computer terminal, which can be any one of the computer terminal devices in the computer terminal group. Alternatively, in the present embodiment, the above-mentioned computer terminal can also be replaced by a mobile terminal or an electronic device or the like terminal device.

[0101] Alternatively, in the present embodiment, the above-mentioned computer terminal can be located in at least one network device of a plurality of network devices of a computer network.

[0102] In the embodiment, the computer terminal can execute program codes of the following steps in the method for detecting atmospheric visibility based on a multi-modal model: obtaining image data and meteorological data, wherein the image data is obtained by a monitoring device deployed on a target device, and the meteorological data is obtained by a preset meteorological device deployed on the target device; generating a preset question based on the image data and the meteorological data; inputting the preset question into a target multi-modal model to output reply information for the preset question, and determining atmospheric visibility at a location of the target device based on the reply information, wherein the target multi-modal model is one of a first multi-modal model and a second multi-modal model, the network structure of the first multi-modal model at least includes a first preset encoder and a time series network structure, and the network structure of the second multi-modal model at least includes a second preset encoder and a feature extraction network.

[0103] Optionally, the computer terminal can execute program codes of the following steps in the method for detecting atmospheric visibility based on a multi-modal model: collecting first historical video data and first historical meteorological data of a plurality of devices within a preset period of time, and frame extracting the first historical video data to obtain a plurality of first historical image data; determining a sequence data set based on all the first historical meteorological data and all the first historical image data, wherein the first historical meteorological data at least includes temperature, humidity, air pressure, wind speed, and atmospheric visibility value; constructing a first question-text pair set based on the sequence data set; and training a first initial multi-modal model based on the first question-text pair set to obtain a first multi-modal model.

[0104] Optionally, the computer terminal can execute program codes of the following steps in the method for detecting atmospheric visibility based on a multi-modal model: collecting second historical image data and second historical meteorological data of a plurality of devices at a plurality of times to obtain a data set; constructing a second question-text pair set based on the data set; and training a second initial multi-modal model based on the second question-text pair set to obtain a second multi-modal model.

[0105] Optionally, the computer terminal can execute program codes of the following steps in the method for detecting atmospheric visibility based on a multi-modal model: for each first historical image data and each first historical meteorological data in the sequence data set, generating a first meteorological question text based on the first historical image data and temperature, humidity, air pressure, and wind speed; generating a first meteorological answer text based on the atmospheric visibility value, wherein the first meteorological answer text is a reply to the first meteorological question text; and constructing a first question-text pair set based on all the first meteorological question texts and all the first meteorological answer texts.

[0106] Optionally, the computer terminal can execute program codes of the following steps in the method for detecting atmospheric visibility based on a multi-modal model: generating second meteorological question texts based on each second historical image data in the data set and temperature, humidity, air pressure and wind speed in each second historical meteorological data; generating second meteorological answer texts based on the atmospheric visibility numerical value; and constructing a second question pair text set based on all second meteorological question texts and all second meteorological answer texts.

[0107] Optionally, the computer terminal can execute program codes of the following steps in the method for detecting atmospheric visibility based on a multi-modal model: constructing a first preset encoder, wherein the first preset encoder is used to extract features of each first historical image data to obtain first image features, and a plurality of fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded into a first preset position of a corresponding first meteorological question text in the first question pair text set; constructing a time series network structure, wherein the time series network structure is used to extract features of each first historical meteorological data to obtain first target time series features, and the first target time series features are embedded into a second preset position in the first meteorological question text; constructing a first initial multi-modal model based on the first preset encoder, the time series network structure and a preset multi-modal model; constructing a first preset feature vector, and embedding the first preset feature vector into a third preset position of the first meteorological question text and a fourth preset position of the first meteorological question text to obtain a first target question pair text set; and training the first initial multi-modal model based on the first target question pair text set based on a preset training strategy to obtain a first multi-modal model.

[0108] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the atmospheric visibility detection method based on a multimodal model: constructing a second preset encoder, wherein the second preset encoder is used to extract features from each second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded in the first preset position of the corresponding second meteorological question text in the second question text set; constructing a feature extraction network, wherein the feature extraction network is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded in the second preset position of the second meteorological question text; constructing a second initial multimodal model based on the second preset encoder, the feature extraction network, and the preset multimodal model; constructing a second preset feature vector, and embedding the second preset feature vector in the fifth preset position of the second meteorological question text, and embedding the second preset feature vector in the sixth preset position of the second meteorological question text to obtain the second target question text set; and training the second initial multimodal model using the second target question text set based on a preset training strategy to obtain the second multimodal model.

[0109] Optionally, Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 9 As shown, the electronic device may include: one or more ( Figure 9 (Only one is shown) processor 902, memory 904, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0110] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the atmospheric visibility detection method and device based on a multimodal model in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned atmospheric visibility detection method based on a multimodal model. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The processor can access the information and application programs stored in the memory via the transmission device to execute the steps described above in the atmospheric visibility detection method based on the multimodal model.

[0112] By adopting the embodiment of the application, a scheme for detecting atmospheric visibility based on a multi-modal model is provided. By constructing a text set of question-answer pairs and performing multi-modal model training based on the text set, deep features of image data and meteorological data such as temperature and humidity, air pressure, and wind speed can be converted into inputs that can be understood by the model, accurate prediction of atmospheric visibility is achieved, and by adding a preset feature vector (such as a learnable prompt word) before and after the question, the context understanding and complex logical reasoning ability of the model can be enhanced, so that the model can still maintain robustness and high precision when facing diversified meteorological environments, thereby solving the technical problem of low accuracy of atmospheric visibility detection in the related art.

[0113] Those skilled in the art can understand that Figure 9 The structure shown is only schematic, and the electronic device can also be a terminal device such as a smart phone, a tablet computer, a palm computer, and a mobile Internet device (MID). Figure 9 It does not limit the structure of the electronic device. For example, the electronic device can include more or fewer components (such as a network interface, a display device, etc.) than Figure 9 or have a different configuration than Figure 9 shown.

[0114] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing the related hardware of the terminal device, and the programs can be stored in a computer-readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0115] Embodiment 4

[0116] The embodiment of the application further provides a storage medium. Optionally, in the embodiment, the above storage medium can be used to save program codes executed by the method for detecting atmospheric visibility based on a multi-modal model provided in the embodiment one.

[0117] Optionally, in the embodiment, the above storage medium can be located in any one of computer terminals in a computer terminal group in a computer network, or in any one of mobile terminals in a mobile terminal group.

[0118] The application further provides a computer program product adapted to execute the steps of the method for detecting atmospheric visibility based on a multi-modal model when executed on a data processing device.

[0119] The above embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0120] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0121] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic, for example, the division of units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0122] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0123] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0124] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0125] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.< / xxx> < / xxx> < / xxx> < / xxx> < / vis> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx> ​

Claims

1. A method for detecting atmospheric visibility based on a multimodal model, characterized in that, include: Acquire image data and meteorological data, wherein the image data is obtained through monitoring equipment deployed on the target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device; Based on the image data and the meteorological data, generate a preset question; The preset question is input into the target multimodal model, which outputs the response information to the preset question. Based on the response information, the atmospheric visibility at the location of the target device is determined. The target multimodal model is one of a first multimodal model and a second multimodal model. The network structure of the first multimodal model includes at least a first preset encoder and a time series network structure. The network structure of the second multimodal model includes at least a second preset encoder and a feature extraction network.

2. The atmospheric visibility detection method based on a multimodal model according to claim 1, characterized in that, Before inputting the preset question into the target multimodal model and outputting the response information for the preset question, the method further includes: Collect first historical video data and first historical meteorological data from multiple devices within a preset time period, and extract frames from the first historical video data to obtain multiple first historical image data. Based on all the first historical meteorological data and all the first historical image data, a sequence data set is determined, wherein the first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed and atmospheric visibility values. Based on the sequence data set, construct the first question pair text set; Based on the first question and the text set, the first initial multimodal model is trained to obtain the first multimodal model.

3. The atmospheric visibility detection method based on a multimodal model according to claim 1, characterized in that, Before inputting the preset question into the target multimodal model and outputting the response information for the preset question, the method further includes: Second historical image data and second historical meteorological data from multiple devices at multiple times were collected to obtain a dataset; Based on the aforementioned data set, a second set of question-pair texts is constructed; Based on the second question and the text set, the second initial multimodal model is trained to obtain the second multimodal model.

4. The atmospheric visibility detection method based on a multimodal model according to claim 2, characterized in that, The steps for constructing the first question pair text set based on the sequence data set include: For each first historical image data and each first historical meteorological data in the sequence data set, a first meteorological question text is generated based on the first historical image data and the temperature, humidity, air pressure and wind speed; Based on the atmospheric visibility value, a first meteorological response text is generated, wherein the first meteorological response text is a reply to the first meteorological question text; Based on all the first meteorological question texts and all the first meteorological answer texts, construct the first question-and-answer text set.

5. The atmospheric visibility detection method based on a multimodal model according to claim 3, characterized in that, The steps for constructing the second question-pair text set based on the aforementioned data set include: Based on the temperature, humidity, air pressure, and wind speed in each of the second historical image data and each of the second historical meteorological data in the dataset, a second meteorological question text is generated; Generate a second meteorological response text based on atmospheric visibility values; Based on all the second meteorological question texts and all the second meteorological answer texts, construct the second question pair text set.

6. The atmospheric visibility detection method based on a multimodal model according to claim 2, characterized in that, The steps for training a first initial multimodal model based on the first question set of texts to obtain the first multimodal model include: Construct the first preset encoder, wherein the first preset encoder is used to extract features from each of the first historical image data to obtain first image features, and multiple fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded in the first preset position of the first meteorological question text corresponding to the first question text set. Construct the time series network structure, wherein the time series network structure is used to extract features from each of the first historical meteorological data to obtain a first target time series feature, and the first target time series feature is embedded in a second preset position in the first meteorological question text; Based on the first preset encoder, the time series network structure, and the preset multimodal model, the first initial multimodal model is constructed; Construct a first preset feature vector, embed the first preset feature vector at a third preset position in the first meteorological question text, and embed the first preset feature vector at a fourth preset position in the first meteorological question text to obtain a first target question pair text set; Based on a preset training strategy, the first initial multimodal model is trained using the first target question text set to obtain the first multimodal model.

7. The atmospheric visibility detection method based on a multimodal model according to claim 3, characterized in that, The steps for training a second initial multimodal model on a text set based on the second problem to obtain the second multimodal model include: Construct the second preset encoder, wherein the second preset encoder is used to extract features from each of the second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into the first preset position of the corresponding second meteorological question text in the second question text set; The feature extraction network is constructed, wherein the feature extraction network is used to extract features from each of the second historical meteorological data to obtain a second target meteorological feature, and the second target meteorological feature is embedded in a second preset position in the second meteorological question text; Based on the second preset encoder, the feature extraction network, and the preset multimodal model, the second initial multimodal model is constructed; Construct a second preset feature vector, and embed the second preset feature vector at the fifth preset position of the second meteorological question text, and embed the second preset feature vector at the sixth preset position of the second meteorological question text to obtain the second target question pair text set; Based on a preset training strategy, the second initial multimodal model is trained using the second target question text set to obtain the second multimodal model.

8. A device for detecting atmospheric visibility based on a multimodal model, characterized in that, include: An acquisition unit is used to acquire image data and meteorological data, wherein the image data is obtained through monitoring equipment deployed on the target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device; The generation unit is used to generate a preset question based on the image data and the meteorological data; A determining unit is configured to input the preset question into a target multimodal model, output response information to the preset question, and determine the atmospheric visibility at the location of the target device based on the response information. The target multimodal model is one of a first multimodal model and a second multimodal model. The network structure of the first multimodal model includes at least a first preset encoder and a time series network structure, and the network structure of the second multimodal model includes at least a second preset encoder and a feature extraction network.

9. A computer program product, characterized in that, The method includes a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the atmospheric visibility detection method based on a multimodal model as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the atmospheric visibility detection method based on a multimodal model as described in any one of claims 1 to 7.

Images