Method and device for detecting atmospheric visibility based on a multi-modal model

Abstract

The application discloses a kind of detection method and device of atmospheric visibility based on multi-modal model, it is related to artificial intelligence technical field, wherein, the detection method includes: obtaining image data and meteorological data;Based on image data and meteorological data, generate preset question;The preset question is input to target multi-modal model, and the reply information for preset question is output, and based on reply information, the atmospheric visibility of the location where target device is located is determined.The application solves the technical problem that the accuracy of atmospheric visibility detection in the related art is low.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for detecting atmospheric visibility based on a multimodal model. Background Technology

[0002] With the development of image processing and deep learning technologies, the visibility estimation methods commonly used in related technologies mainly include: dark channel prior theory, structural similarity, deep neural networks, etc., which indirectly infer visibility by analyzing image features (such as brightness, contrast, color saturation, etc.).

[0003] However, current atmospheric visibility analysis methods face challenges such as limited data sources, poor model adaptability, insufficient analytical accuracy, and difficulty in coping with complex meteorological environments.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method and apparatus for detecting atmospheric visibility based on a multimodal model, which at least solves the technical problem of low accuracy in atmospheric visibility detection in related technologies.

[0006] According to one aspect of the embodiments of this application, a method for detecting atmospheric visibility based on a multimodal model is provided, comprising: acquiring image data and meteorological data, wherein the image data is obtained through monitoring equipment deployed on a target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device; generating a preset question based on the image data and meteorological data; inputting the preset question into a target multimodal model, outputting response information to the preset question, and determining the atmospheric visibility at the location of the target device based on the response information, wherein the target multimodal model is one of a first multimodal model and a second multimodal model, the network structure of the first multimodal model includes at least: a first preset encoder and a time series network structure, and the network structure of the second multimodal model includes at least: a second preset encoder and a feature extraction network.

[0007] Furthermore, before inputting the preset question into the target multimodal model and outputting the response information for the preset question, the method further includes: collecting first historical video data and first historical meteorological data from multiple devices within a preset time period, and extracting frames from the first historical video data to obtain multiple first historical image data; determining a sequence data set based on all the first historical meteorological data and all the first historical image data, wherein the first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed, and atmospheric visibility values; constructing a first question-to-text set based on the sequence data set; and training the first initial multimodal model based on the first question-to-text set to obtain the first multimodal model.

[0008] Furthermore, before inputting the preset question into the target multimodal model and outputting the response information for the preset question, the method further includes: collecting second historical image data and second historical meteorological data from multiple devices at multiple times to obtain a dataset; constructing a second question-text set based on the dataset; and training the second initial multimodal model based on the second question-text set to obtain the second multimodal model.

[0009] Furthermore, the step of constructing a first question-and-answer text set based on the sequence data set includes: for each first historical image data and each first historical meteorological data in the sequence data set, generating a first meteorological question text based on the first historical image data and temperature, humidity, air pressure, and wind speed; generating a first meteorological answer text based on atmospheric visibility values, wherein the first meteorological answer text is a response to the first meteorological question text; and constructing a first question-and-answer text set based on all first meteorological question texts and all first meteorological answer texts.

[0010] Furthermore, the step of constructing a second question-and-answer text set based on the dataset includes: generating a second meteorological question text based on the temperature, humidity, air pressure, and wind speed in each second historical image data and each second historical meteorological data in the dataset; generating a second meteorological answer text based on the atmospheric visibility value; and constructing a second question-and-answer text set based on all second meteorological question texts and all second meteorological answer texts.

[0011] Further, the step of training a first initial multimodal model based on a first question pair text set to obtain a first multimodal model includes: constructing a first preset encoder, wherein the first preset encoder is used to extract features from each first historical image data to obtain first image features, and multiple fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded in a first preset position in the corresponding first meteorological question text in the first question pair text set; constructing a time series network structure, wherein the time series network structure is used to extract features from each first historical meteorological data to obtain first target time series features, and the first target time series features are embedded in a second preset position in the first meteorological question text; constructing a first initial multimodal model based on the first preset encoder, the time series network structure, and the preset multimodal model; constructing a first preset feature vector, and embedding the first preset feature vector in a third preset position in the first meteorological question text, and embedding the first preset feature vector in a fourth preset position in the first meteorological question text to obtain a first target question pair text set; and training the first initial multimodal model using the first target question pair text set based on a preset training strategy to obtain the first multimodal model.

[0012] Further, the step of training the second initial multimodal model based on the second question pair text set to obtain the second multimodal model includes: constructing a second preset encoder, wherein the second preset encoder is used to extract features from each second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into the first preset position of the corresponding second meteorological question text in the second question pair text set; constructing a feature extraction network, wherein the feature extraction network is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into the second preset position in the second meteorological question text; constructing a second initial multimodal model based on the second preset encoder, the feature extraction network, and the preset multimodal model; constructing a second preset feature vector, and embedding the second preset feature vector into the fifth preset position of the second meteorological question text, and embedding the second preset feature vector into the sixth preset position of the second meteorological question text to obtain the second target question pair text set; and training the second initial multimodal model using the second target question pair text set based on a preset training strategy to obtain the second multimodal model.

[0013] According to another aspect of the embodiments of this application, an atmospheric visibility detection device based on a multimodal model is also provided, comprising: an acquisition unit for acquiring image data and meteorological data, wherein the image data is obtained through a monitoring device deployed on a target device, and the meteorological data is obtained through a preset meteorological device deployed on the target device; a generation unit for generating a preset question based on the image data and the meteorological data; and a determination unit for inputting the preset question into a target multimodal model, outputting response information to the preset question, and determining the atmospheric visibility at the location of the target device based on the response information, wherein the target multimodal model is one of a first multimodal model and a second multimodal model, the network structure of the first multimodal model includes at least a first preset encoder and a time series network structure, and the network structure of the second multimodal model includes at least a second preset encoder and a feature extraction network.

[0014] Furthermore, the atmospheric visibility detection device based on a multimodal model also includes: a first acquisition module, used to acquire first historical video data and first historical meteorological data of multiple devices within a preset time period before inputting a preset question into the target multimodal model and outputting response information for the preset question, and to extract frames from the first historical video data to obtain multiple first historical image data; a first determination module, used to determine a sequence data set based on all first historical meteorological data and all first historical image data, wherein the first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed and atmospheric visibility values; a first construction module, used to construct a first question-pair text set based on the sequence data set; and a first training module, used to train a first initial multimodal model based on the first question-pair text set to obtain a first multimodal model.

[0015] Furthermore, the atmospheric visibility detection device based on the multimodal model also includes: a second acquisition module, used to acquire second historical image data and second historical meteorological data of multiple devices at multiple times before inputting the preset question into the target multimodal model and outputting the response information to the preset question, to obtain a data set; a second construction module, used to construct a second question-pair text set based on the data set; and a second training module, used to train the second initial multimodal model based on the second question-pair text set to obtain the second multimodal model.

[0016] Furthermore, the first construction module includes: a first generation submodule, used to generate a first meteorological question text based on the first historical image data and temperature, humidity, air pressure and wind speed for each first historical image data and each first historical meteorological data in the sequence data set; a second generation submodule, used to generate a first meteorological response text based on atmospheric visibility values, wherein the first meteorological response text is a reply to the first meteorological question text; and a first construction submodule, used to construct a first question-and-answer text set based on all first meteorological question texts and all first meteorological response texts.

[0017] Furthermore, the second construction module includes: a third generation submodule, used to generate second meteorological question text based on the temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the dataset; a fourth generation submodule, used to generate second meteorological answer text based on atmospheric visibility values; and a second construction submodule, used to construct a second question-and-answer text set based on all second meteorological question texts and all second meteorological answer texts.

[0018] Further, the first training module includes: a third construction submodule, used to construct a first preset encoder, wherein the first preset encoder is used to extract features from each first historical image data to obtain first image features, and multiple fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded into a first preset position in the first meteorological question text corresponding to the first question text set; a fourth construction submodule, used to construct a time series network structure, wherein the time series network structure is used to extract features from each first historical meteorological data to obtain first target time series features, and the first target time series features are embedded into the first preset position in the first question text set. The system includes: a second preset position embedded in the first meteorological question text; a fifth construction submodule for constructing a first initial multimodal model based on a first preset encoder, a time series network structure, and a preset multimodal model; a sixth construction submodule for constructing a first preset feature vector and embedding the first preset feature vector at a third preset position and a fourth preset position in the first meteorological question text to obtain a first target question pair text set; and a first training submodule for training the first initial multimodal model using the first target question pair text set based on a preset training strategy to obtain the first multimodal model.

[0019] Further, the second training module includes: a seventh construction submodule, used to construct a second preset encoder, wherein the second preset encoder is used to extract features from each second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into the first preset position of the corresponding second meteorological question text in the second question text set; an eighth construction submodule, used to construct a feature extraction network, wherein the feature extraction network is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into the second preset position in the second meteorological question text; a ninth construction submodule, used to construct a second initial multimodal model based on the second preset encoder, the feature extraction network, and the preset multimodal model; a tenth construction submodule, used to construct a second preset feature vector, and embed the second preset feature vector into the fifth preset position of the second meteorological question text, and embed the second preset feature vector into the sixth preset position of the second meteorological question text to obtain the second target question text set; and a second training submodule, used to train the second initial multimodal model using the second target question text set based on a preset training strategy to obtain the second multimodal model.

[0020] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements any one of the above-described methods for detecting atmospheric visibility based on a multimodal model.

[0021] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described methods for detecting atmospheric visibility based on a multimodal model.

[0022] In this invention, image data and meteorological data are acquired; a preset question is generated based on the image data and meteorological data; the preset question is input into a target multimodal model, and response information for the preset question is output. Based on the response information, the atmospheric visibility at the location of the target device is determined, thus solving the technical problem of low accuracy in atmospheric visibility detection in related technologies.

[0023] In this invention, real-time images and meteorological data are acquired from monitoring equipment on the target device and a preset weather station. Based on the image data and meteorological data, a preset question is generated and input into a target multimodal model. This target multimodal model can be a first multimodal model, whose network structure includes at least a first preset encoder for processing multi-time-series image data and a time-series network for integrating meteorological time-series data. Alternatively, it can be a second multimodal model, whose network structure includes at least a second preset encoder for processing single-time-series image data and a feature extraction network for extracting meteorological features. Through analysis by the target multimodal model, response information about atmospheric visibility can be output. Based on the response information, the atmospheric visibility at the location of the target device can be determined, improving the accuracy of atmospheric visibility prediction under complex meteorological conditions. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0025] Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a method for detecting atmospheric visibility based on a multimodal model is shown.

[0026] Figure 2 This is a flowchart of an atmospheric visibility detection method based on a multimodal model according to Embodiment 1 of this application;

[0027] Figure 3 This is a schematic diagram of an optional question-and-answer pair according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of an optional multimodal large model inference according to an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of an optional multimodal large model structure utilizing multiple time-series images and meteorological data according to an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of an optional multimodal large model structure utilizing a single time-series image and meteorological data according to an embodiment of this application;

[0031] Figure 7 This is a flowchart of an optional atmospheric visibility prediction according to an embodiment of this application;

[0032] Figure 8 This is a schematic diagram of an optional multimodal model-based atmospheric visibility detection device according to an embodiment of this application;

[0033] Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] It should be noted that all related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected and involved in this invention are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and it does not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0037] In this invention, during the data processing stage, images and meteorological data such as temperature, humidity, air pressure, and wind speed are processed in a targeted manner to generate question-answer pairs to meet the input requirements of the multimodal large model. During the multimodal large model training stage, two types of multimodal large models are constructed: one uses multi-time-series images and meteorological data, which has better accuracy, and the other uses single-time-series images and meteorological data, which has slightly lower accuracy but faster data processing and training speed. During the multimodal large model inference stage, multimodal data such as images and meteorological data collected by equipment mounted on the tower are used as input to the multimodal large model to ensure that atmospheric visibility analysis results can be output quickly and accurately under complex and changeable meteorological conditions.

[0038] The present invention will now be described in detail with reference to various embodiments.

[0039] Example 1

[0040] According to an embodiment of this application, an embodiment of an atmospheric visibility detection method based on a multimodal model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a method for detecting atmospheric visibility based on a multimodal model is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions may also be included. In addition, it may include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera, wherein the network interface can be connected to wired and / or wireless networks. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0042] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the atmospheric visibility detection method based on a multimodal model in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned atmospheric visibility detection method based on a multimodal model. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0045] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0046] Under the aforementioned operating environment, this application provides the following: Figure 2 The method for detecting atmospheric visibility based on a multimodal model is shown. Figure 2 This is a flowchart of the atmospheric visibility detection method based on a multimodal model according to Embodiment 1 of this application, as follows: Figure 2 As shown, the method includes the following steps:

[0047] Step S201: Acquire image data and meteorological data, wherein the image data is obtained through monitoring equipment deployed on the target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device; based on the image data and meteorological data, generate preset questions.

[0048] In this embodiment of the invention, image data collected by monitoring equipment deployed on the target device (such as a communication tower of an iron tower) and meteorological data collected by preset meteorological equipment (such as a micro weather station) deployed on the target device are acquired.

[0049] For example, the communication tower is equipped with a surveillance camera and a mini weather station. The surveillance camera captured a video, while the mini weather station recorded the temperature as 20°C, humidity as 60%, air pressure as 1013 hPa, and wind speed as 5 m / s.

[0050] Step S202: Generate a preset question based on image data and meteorological data.

[0051] In this embodiment of the invention, based on the acquired image data and meteorological data, questions are constructed according to a pre-set template (such as a question-answer pair template), thereby generating a preset question (e.g., based on the following images ( ) and meteorological data (temperature is <xxx>Humidity is <xxx>air pressure is <xxx>Wind speed is <xxx>(Please analyze the current atmospheric visibility in meters.)

[0052] It is important to note that This represents the image data collected from surveillance cameras after feature extraction.

[0053] Step S203: Input the preset question into the target multimodal model, output the response information to the preset question, and determine the atmospheric visibility at the location of the target device based on the response information. The target multimodal model is one of a first multimodal model and a second multimodal model. The network structure of the first multimodal model includes at least a first preset encoder and a time series network structure, and the network structure of the second multimodal model includes at least a second preset encoder and a feature extraction network.

[0054] Figure 3 This is an illustration of an optional question-and-answer format according to an embodiment of this application, such as... Figure 3 As shown, the problem could be: "Atmospheric visibility distance can be determined by analyzing images. Furthermore, temperature, humidity, air pressure, wind speed, etc., are all related to atmospheric visibility distance. The image is currently..." Given the following conditions: temperature xxx, humidity xxx, air pressure xxx, wind speed xxx, please conduct a comprehensive analysis based on the image, temperature, humidity, air pressure, wind speed, etc., and analyze the atmospheric visibility distance of the current scene. A possible answer is: "Based on the comprehensive analysis of the image, temperature, humidity, air pressure, wind speed, etc., the current atmospheric visibility distance is xxx meters."

[0055] In this embodiment of the invention, the target multimodal model refers to the multimodal model after training, which can be a first multimodal model or a second multimodal model. The first multimodal model is trained based on time-series images using a first preset encoder and a time-series network structure, while the second multimodal model is trained based on single-frame images using a second preset encoder and a feature extraction network. By inputting a preset question into the target multimodal model, the model outputs response information to the preset question, and based on the response information, the atmospheric visibility at the location of the target device can be determined (e.g., the atmospheric visibility in the current scene is 1000 meters).

[0056] Figure 4 This is a schematic diagram of an optional multimodal large model inference according to an embodiment of this application, such as... Figure 4 As shown, a monitoring camera and a miniature weather station (i.e., a miniature visibility measurement station) are deployed on the communication tower. Multiple frames of images and multi-time-series meteorological data such as temperature, humidity, air pressure, and wind speed are acquired through the monitoring camera and the miniature weather station. These meteorological factors (i.e., meteorological data) are input into an LSTM (Long Short-Term Memory) model for feature extraction to obtain meteorological features. Simultaneously, multiple frames of images are input into a 3D encoder for feature extraction to obtain image features. These image features are then processed through a fully connected layer. The meteorological features and the processed image features are embedded into the corresponding positions in the question, and learnable prompts are added before and after the question. This information is then input into an LLM (Large Language Model) to answer the question about the atmospheric visibility at the location of the communication tower.

[0057] In summary, the collected images and meteorological data are transformed into a target multimodal model, which can be selected from either a first or second multimodal model. The first multimodal model can handle multiple time-series information and has better accuracy, while the second multimodal model can handle single time-series information and is more lightweight. By selecting and determining the target multimodal model, atmospheric visibility prediction values ​​can be output, overcoming the limitations of single-modal data, making full use of the complementary advantages of images and meteorological data, improving the accuracy and adaptability of visibility analysis, and thus solving the technical problem of low accuracy in atmospheric visibility detection in related technologies.

[0058] To accurately obtain the first multimodal model, in the atmospheric visibility detection method based on the multimodal model provided in Embodiment 1 of this application, first historical video data and first historical meteorological data from multiple devices within a preset time period are collected, and frames are extracted from the first historical video data to obtain multiple first historical image data; based on all the first historical meteorological data and all the first historical image data, a sequence data set is determined, wherein the first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed, and atmospheric visibility values; based on the sequence data set, a first question-and-text set is constructed; based on the first question-and-text set, a first initial multimodal model is trained to obtain the first multimodal model.

[0059] Optionally, the first historical meteorological data may include at least temperature, humidity, air pressure, wind speed, and atmospheric visibility values, and may also include fewer or more meteorological data, such as adding rainfall, PM2.5, etc., or reducing one or more of the following data: temperature, humidity, air pressure, wind speed, etc.

[0060] In this embodiment of the invention, first historical video data and first historical meteorological data from multiple devices within a preset time period (e.g., within one week) are collected, and frames are extracted from the first historical video data to obtain multiple first historical image data. By extracting key frames from the video data, the amount of data processing is reduced, while retaining enough information for feature analysis.

[0061] In this embodiment of the invention, the first historical image data is integrated with the first historical meteorological data at the corresponding time to form a serialized data set (i.e., a sequence data set) so that the model can learn the influence of time series. The first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed, and atmospheric visibility values. Based on the sequence data set, a first question-text set is constructed (e.g., Question: Considering...). and environmental parameters (temperature is <xxx>Humidity is <xxx>air pressure is <xxx>Wind speed is <xxx>Predict the current visibility. Answer: The current visibility is [missing value]. <vis>After that, based on the first question and the text set, the first initial multimodal model is trained to obtain the first multimodal model.

[0062] In order to accurately obtain the second multimodal model, in the atmospheric visibility detection method based on the multimodal model provided in Embodiment 1 of this application, second historical image data and second historical meteorological data of multiple devices at multiple times are collected to obtain a data set; based on the data set, a second question-pair text set is constructed; based on the second question-pair text set, the second initial multimodal model is trained to obtain the second multimodal model.

[0063] In this embodiment of the invention, the first multimodal model uses multi-frame image data and corresponding multi-time series temperature, humidity, air pressure, and wind speed data for model training and inference. Since the use of multi-time series information is more effective, it is also more time-consuming. Therefore, a question pair constructed using a single-frame image and its corresponding single-time series temperature, humidity, air pressure, wind speed, and other meteorological data can be used as training data to train the model. By collecting second historical image data and second historical meteorological data from multiple devices at multiple times (such as different times of day), a data set is obtained. Based on the data set, a second question pair text set can be constructed, and the second question pair text set can be used as training data to train the second initial multimodal model to obtain the second multimodal model.

[0064] In order to accurately construct the first question-and-answer text set, in the atmospheric visibility detection method based on a multimodal model provided in Embodiment 1 of this application, for each first historical image data and each first historical meteorological data in the sequence data set, a first meteorological question text is generated based on the first historical image data and temperature, humidity, air pressure and wind speed; a first meteorological answer text is generated based on the atmospheric visibility value, wherein the first meteorological answer text is a response to the first meteorological question text; and a first question-and-answer text set is constructed based on all first meteorological question texts and all first meteorological answer texts.

[0065] In this embodiment of the invention, the sequence data set includes first historical image data and corresponding first historical meteorological data collected continuously over a period of time. For each first historical image data and each first historical meteorological data in the sequence data set, a first meteorological question text can be generated based on the first historical image data and temperature, humidity, air pressure and wind speed, and a first meteorological answer text (i.e., a response to the first meteorological question text) can be generated based on atmospheric visibility values. Based on all first meteorological question texts and all first meteorological answer texts, a first question-and-answer text set is constructed.

[0066] In order to accurately construct the second question-and-answer text set, in the atmospheric visibility detection method based on a multimodal model provided in Embodiment 1 of this application, a second meteorological question text is generated based on the temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the dataset; a second meteorological answer text is generated based on the atmospheric visibility value; and a second question-and-answer text set is constructed based on all the second meteorological question texts and all the second meteorological answer texts.

[0067] In this embodiment of the invention, the data set refers to the sum of the collected second historical image data and second historical meteorological data. Based on the temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the data set, a second meteorological question text is generated. Based on the atmospheric visibility value, a second meteorological answer text is generated. Based on all the second meteorological question texts and all the second meteorological answer texts, a second question-and-answer text set is constructed.

[0068] To improve the accuracy of the first multimodal model, in the atmospheric visibility detection method based on a multimodal model provided in Embodiment 1 of this application, a first preset encoder is constructed. The first preset encoder is used to extract features from each first historical image data to obtain first image features. Multiple fully connected layers are used to process the first image features to obtain first target image features. The first target image features are then embedded into a first preset position in the corresponding first meteorological question text of a first question-and-answer text set. A time-series network structure is constructed. The time-series network structure is used to extract features from each first historical meteorological data to obtain first target time-series features. These first target time-series features are then embedded into a second preset position in the first meteorological question text. Based on the first preset encoder, the time-series network structure, and the preset multimodal model, a first initial multimodal model is constructed. A first preset feature vector is constructed and embedded into a third preset position and a fourth preset position in the first meteorological question text to obtain a first target question-and-answer text set. Based on a preset training strategy, the first initial multimodal model is trained using the first target question-and-answer text set to obtain the first multimodal model.

[0069] In this embodiment of the invention, a first preset encoder (such as a 3D encoder, based on the Transformer (a neural network architecture) architecture, with a self-attention mechanism for processing video frames and spatial pixels) is constructed. Features are extracted from each first historical image data using the first preset encoder to obtain first image features. Multiple fully connected layers are then used to process these first image features to obtain first target image features (such as tokens). This ensures that the format and dimensions of these features are adapted to the model's input requirements. Furthermore, the first target image features can be embedded into a first preset position in the first meteorological question text corresponding to the first question-and-answer text set (e.g., ...). (Location). For example, the first question text could be: Please analyze the following image features ([token1, token2, ..., tokenN]) and environmental parameters to predict visibility.

[0070] Next, a time-series network structure is constructed. Features are extracted from each first historical meteorological data point using a time-series network structure (such as an LSTM network, which is not limited here), yielding the time-series features of the meteorological data (i.e., the first target time-series features). These first target time-series features are then embedded into a second preset position within the first meteorological question text (e.g., ...). <xxx>(Location), for example, the first question text in the first meteorological question text could be: Please analyze the following image features ([token1, token2, ..., tokenN]) and environmental parameters ([seq_temp, seq_hum, ..., seq_wind]) to predict visibility. Furthermore, a first initial multimodal model can be constructed based on a first preset encoder, a time-series network structure, and a preset multimodal model (such as large models like QwenLlama, deepseek, and Intern).

[0071] In this embodiment of the invention, a first preset feature vector (i.e., learnable prompt words) is constructed, and the first preset feature vector is embedded at a third preset position (e.g., the beginning of the first weather question text) and at a fourth preset position (e.g., the end of the first weather question text), to obtain a first target question pair text set. Based on a preset training strategy, the first target question pair text set is used to train a first initial multimodal model to obtain a first multimodal model. For example, two sets of learnable tokens (i.e., the first preset feature vector, such as...) are added at the beginning and end of the first weather question text, respectively.<start_prompt> ,<end_prompt> ), forming the first target question set of texts: "<start_prompt> Please analyze the following image features ([token1, token2, ..., tokenN]) and environmental parameters ([seq_temp, seq_hum, ..., seq_wind]) to predict visibility.<end_prompt> ".

[0072] Subsequently, based on a pre-defined training strategy (such as next-token-prediction, a supervised learning method), the first initial multimodal model is trained on the text set using the first objective question to obtain the first multimodal model.

[0073] Figure 5 This is a schematic diagram of an optional multimodal large model structure utilizing multiple time-series images and meteorological data according to an embodiment of this application. The multimodal large model structure includes an LLM (i.e., a preset multimodal model), a 3D encoder (i.e., a first preset encoder), an LSTM (i.e., a time series network structure), and a Connector (i.e., a fully connected layer). First, the 3D encoder is used to extract features from multiple frames of images extracted from a video. The 3D encoder is composed of Transformers. After the extracted features are in token form, the Connector, composed of multiple fully connected layers, performs feature transformation to align with the input of the LLM. Finally, the output tokens are embedded into a question-and-answer matching problem. Location, and simultaneously, after normalizing the meteorological factors such as temperature, humidity, air pressure, and wind speed corresponding to multiple frames of images, they are input into an LSTM network to extract features from multi-time-series temperature, humidity, air pressure, and wind speed data. Finally, the extracted corresponding features are embedded into the question-answer matching problem. <xxx>After determining the location, the extracted multi-frame image features are integrated with temperature, humidity, air pressure, and wind speed features into the question. Learnable prompts are added before and after the question, and the model is trained based on the answers (i.e., the first target question set of texts). During the training process, the LLM is also fine-tuned to obtain a multimodal large model (i.e., the first multimodal model).

[0074] To improve the accuracy of the second multimodal model, in the atmospheric visibility detection method based on the multimodal model provided in Embodiment 1 of this application, a second preset encoder is constructed. The second preset encoder extracts features from each second historical image data to obtain second image features. Multiple fully connected layers process the second image features to obtain second target image features. These second target image features are then embedded into the first preset position of the corresponding second meteorological question text in the second question text set. A feature extraction network is constructed. The feature extraction network extracts features from each second historical meteorological data to obtain second target meteorological features, which are then embedded into the second preset position in the second meteorological question text. Based on the second preset encoder, the feature extraction network, and the preset multimodal model, a second initial multimodal model is constructed. A second preset feature vector is constructed and embedded into the fifth preset position and the sixth preset position of the second meteorological question text to obtain the second target question text set. Based on a preset training strategy, the second initial multimodal model is trained using the second target question text set to obtain the second multimodal model.

[0075] In this embodiment of the invention, a second preset encoder (a predefined encoder structure, which can be based on the Transformer architecture, used to process static image data) is constructed. Features are extracted from each second historical image data using the second preset encoder to obtain second image features. Multiple fully connected layers are then used to process these second image features to obtain second target image features. These second target image features are then embedded into a first preset position in the corresponding second meteorological question text within the second question-and-answer text set (e.g., ...). Location).

[0076] Next, a feature extraction network is constructed to extract features from each second historical meteorological data point, obtaining the second target meteorological features. These second target meteorological features are then embedded into a second preset location within the second meteorological question text (e.g., ...). <xxx>(Location), based on the second preset encoder, feature extraction network and preset multimodal model, construct the second initial multimodal model.

[0077] In this embodiment of the invention, a second preset feature vector (i.e., learnable prompt words used to enhance the model's understanding and contextual reasoning abilities) is constructed, and the second preset feature vector is embedded at the fifth preset position (i.e., the beginning of the second meteorological question text) and at the sixth preset position (i.e., the end of the second meteorological question text) of the second meteorological question text to obtain a second target question pair text set. Based on a preset training strategy, the second target question pair text set is used to train the second initial multimodal model to obtain the second multimodal model.

[0078] Figure 6 This is a schematic diagram of an optional multimodal large model structure utilizing single time-series images and meteorological data according to an embodiment of this application, such as... Figure 6 As shown, the multimodal large model structure includes an LLM (i.e., a pre-defined multimodal model), an encoder (i.e., a second pre-defined encoder), a Transformer, and a Connector (i.e., fully connected layers). First, the encoder, composed of Transformers, extracts features from the image. After extracting the features as tokens, a Connector, composed of multiple fully connected layers, performs feature transformation to align with the LLM input. Finally, the output tokens are embedded into the question-answer matching problem. The location is determined by normalizing the meteorological factors such as temperature, humidity, air pressure, and wind speed corresponding to the image. These parameters are then input into a feature extraction network composed of Transformers to extract features from the temperature, humidity, air pressure, and wind speed data. Finally, the extracted features are embedded into the question-answer matching problem. <xxx>After determining the location, the extracted image features are integrated with temperature, humidity, air pressure, and wind speed features into the question, and learnable prompt words are added before and after the question. The model is trained based on the answers (i.e., the second target question set of texts). During the training process, the LLM is also fine-tuned to obtain a multimodal large model (i.e., the second multimodal model).

[0079] Figure 7 This is a flowchart of an optional atmospheric visibility prediction method according to an embodiment of this application, such as... Figure 7 As shown, images are collected by a camera mounted on a communication tower, while meteorological data such as temperature, humidity, air pressure, wind speed, and atmospheric visibility are collected at a weather station near the tower. The collected images and meteorological data are used to construct question-answer pairs, resulting in a question-answer pair dataset. This dataset is then used as training data for a multimodal large model to obtain the target multimodal model. Subsequently, images and meteorological data collected by the communication tower-mounted equipment (i.e., the camera and the miniature weather station) are used to construct a pre-defined question, which is then used as input to the target multimodal large model for inference to obtain the predicted atmospheric visibility value.

[0080] The atmospheric visibility detection method based on a multimodal model provided in this application achieves highly accurate prediction and real-time analysis of atmospheric visibility by combining the construction of a multimodal question-pair text set with the training of a large multimodal model. First, deep features of image and meteorological data are extracted and transformed into the question description portion of the question-pair text set. Then, a target question-pair text set is constructed by embedding target image features and time-series meteorological features into the question text and adding preset feature vectors before and after the questions. Next, using a preset training strategy, the initial multimodal model is trained with the target question-pair text set, resulting in two multimodal models capable of handling single-time-series and multi-time-series data, respectively. By effectively integrating image and meteorological data with the training and fine-tuning of the large multimodal model, atmospheric visibility can be accurately estimated based on real-time image and meteorological data, improving the accuracy and scene adaptability of visibility prediction.

[0081] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0082] Example 2

[0083] This application also provides an atmospheric visibility detection device based on a multimodal model. It should be noted that the atmospheric visibility detection device based on a multimodal model provided in this application can be used to execute the atmospheric visibility detection method based on a multimodal model provided in this application. The atmospheric visibility detection device based on a multimodal model provided in this application will be described below.

[0084] According to an embodiment of this application, an apparatus for implementing the above-described method for detecting atmospheric visibility based on a multimodal model is also provided. Figure 8 This is a schematic diagram of an optional atmospheric visibility detection device based on a multimodal model according to an embodiment of this application, as shown below. Figure 8 As shown, the atmospheric visibility detection device based on a multimodal model may include: an acquisition unit 80, a generation unit 81, and a determination unit 82.

[0085] The acquisition unit 80 is used to acquire image data and meteorological data. The image data is obtained through monitoring equipment deployed on the target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device.

[0086] The generation unit 81 is used to generate a preset question based on image data and meteorological data;

[0087] The determining unit 82 is used to input a preset question into the target multimodal model, output the response information to the preset question, and determine the atmospheric visibility at the location of the target device based on the response information. The target multimodal model is one of a first multimodal model and a second multimodal model. The network structure of the first multimodal model includes at least a first preset encoder and a time series network structure, and the network structure of the second multimodal model includes at least a second preset encoder and a feature extraction network.

[0088] The atmospheric visibility detection device based on a multimodal model provided in this application embodiment can acquire image data and meteorological data through the acquisition unit 80, generate a preset question based on the image data and meteorological data through the generation unit 81, input the preset question into the target multimodal model through the determination unit 82, output the response information to the preset question, and determine the atmospheric visibility at the location of the target device based on the response information.

[0089] Optionally, the atmospheric visibility detection device based on a multimodal model further includes: a first acquisition module, used to acquire first historical video data and first historical meteorological data of multiple devices within a preset time period before inputting a preset question into the target multimodal model and outputting response information for the preset question, and to extract frames from the first historical video data to obtain multiple first historical image data; a first determination module, used to determine a sequence data set based on all first historical meteorological data and all first historical image data, wherein the first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed and atmospheric visibility values; a first construction module, used to construct a first question-pair text set based on the sequence data set; and a first training module, used to train a first initial multimodal model based on the first question-pair text set to obtain a first multimodal model.

[0090] Optionally, the atmospheric visibility detection device based on the multimodal model further includes: a second acquisition module, used to acquire second historical image data and second historical meteorological data of multiple devices at multiple times before inputting a preset question into the target multimodal model and outputting the response information for the preset question, to obtain a data set; a second construction module, used to construct a second question-pair text set based on the data set; and a second training module, used to train the second initial multimodal model based on the second question-pair text set to obtain the second multimodal model.

[0091] Optionally, the first construction module includes: a first generation submodule, used to generate a first meteorological question text based on the first historical image data and temperature, humidity, air pressure and wind speed for each first historical image data and each first historical meteorological data in the sequence data set; a second generation submodule, used to generate a first meteorological response text based on atmospheric visibility values, wherein the first meteorological response text is a reply to the first meteorological question text; and a first construction submodule, used to construct a first question-and-answer text set based on all first meteorological question texts and all first meteorological response texts.

[0092] Optionally, the second construction module includes: a third generation submodule, used to generate second meteorological question text based on temperature, humidity, air pressure and wind speed in each second historical image data and each second historical meteorological data in the dataset; a fourth generation submodule, used to generate second meteorological answer text based on atmospheric visibility values; and a second construction submodule, used to construct a second question-and-answer text set based on all second meteorological question texts and all second meteorological answer texts.

[0093] Optionally, the first training module includes: a third construction submodule for constructing a first preset encoder, wherein the first preset encoder is used to extract features from each first historical image data to obtain first image features, and multiple fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded into a first preset position in the first meteorological question text corresponding to the first question text set; and a fourth construction submodule for constructing a time series network structure, wherein the time series network structure is used to extract features from each first historical meteorological data to obtain first target time series features, and the first target time series features are embedded into a first preset position in the first question text set. The system includes: a second preset position embedded in the first meteorological question text; a fifth construction submodule for constructing a first initial multimodal model based on a first preset encoder, a time series network structure, and a preset multimodal model; a sixth construction submodule for constructing a first preset feature vector and embedding the first preset feature vector at a third preset position and a fourth preset position in the first meteorological question text to obtain a first target question pair text set; and a first training submodule for training the first initial multimodal model using the first target question pair text set based on a preset training strategy to obtain the first multimodal model.

[0094] Optionally, the second training module includes: a seventh construction submodule for constructing a second preset encoder, wherein the second preset encoder is used to extract features from each second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into the first preset position of the corresponding second meteorological question text in the second question text set; an eighth construction submodule for constructing a feature extraction network, wherein the feature extraction network is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into the second preset position in the second meteorological question text; a ninth construction submodule for constructing a second initial multimodal model based on the second preset encoder, the feature extraction network, and the preset multimodal model; a tenth construction submodule for constructing a second preset feature vector, embedding the second preset feature vector into the fifth preset position of the second meteorological question text, and embedding the second preset feature vector into the sixth preset position of the second meteorological question text to obtain the second target question text set; and a second training submodule for training the second initial multimodal model using the second target question text set based on a preset training strategy to obtain the second multimodal model.

[0095] The aforementioned atmospheric visibility detection device based on a multimodal model may also include a processor and a memory. The aforementioned acquisition unit 80, generation unit 81, determination unit 82, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0096] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, a preset question is input into the target multimodal model, which outputs response information to the preset question and, based on the response information, determines the atmospheric visibility at the target device's location.

[0097] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0098] It should be noted that the aforementioned acquisition unit 80, generation unit 81, and determination unit 82 correspond to steps S201 to S203 in Embodiment 1. The instances and application scenarios implemented by these units and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these units can be hardware or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). These units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0099] Example 3

[0100] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced with a mobile terminal or an electronic device, etc.

[0101] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0102] In this embodiment, the aforementioned computer terminal can execute program code for the following steps in the atmospheric visibility detection method based on a multimodal model: acquiring image data and meteorological data, wherein the image data is obtained through monitoring equipment deployed on the target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device; generating a preset question based on the image data and meteorological data; inputting the preset question into the target multimodal model, outputting response information to the preset question, and determining the atmospheric visibility at the location of the target device based on the response information, wherein the target multimodal model is one of a first multimodal model and a second multimodal model, the network structure of the first multimodal model includes at least: a first preset encoder and a time series network structure, and the network structure of the second multimodal model includes at least: a second preset encoder and a feature extraction network.

[0103] Optionally, the aforementioned computer terminal can execute program code for the following steps in the atmospheric visibility detection method based on a multimodal model: collecting first historical video data and first historical meteorological data from multiple devices within a preset time period, and extracting frames from the first historical video data to obtain multiple first historical image data; determining a sequence data set based on all first historical meteorological data and all first historical image data, wherein the first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed, and atmospheric visibility values; constructing a first question-and-text set based on the sequence data set; and training a first initial multimodal model based on the first question-and-text set to obtain a first multimodal model.

[0104] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the atmospheric visibility detection method based on a multimodal model: collecting second historical image data and second historical meteorological data from multiple devices at multiple times to obtain a data set; constructing a second question-pair text set based on the data set; and training a second initial multimodal model based on the second question-pair text set to obtain a second multimodal model.

[0105] Optionally, the aforementioned computer terminal can execute program code for the following steps in the atmospheric visibility detection method based on a multimodal model: for each first historical image data and each first historical meteorological data in the sequence data set, generate a first meteorological question text based on the first historical image data and temperature, humidity, air pressure, and wind speed; generate a first meteorological response text based on atmospheric visibility values, wherein the first meteorological response text is a reply to the first meteorological question text; and construct a first question-and-answer text set based on all first meteorological question texts and all first meteorological response texts.

[0106] Optionally, the aforementioned computer terminal can execute program code for the following steps in the atmospheric visibility detection method based on a multimodal model: generating a second meteorological question text based on the temperature, humidity, air pressure, and wind speed in each second historical image data and each second historical meteorological data in the dataset; generating a second meteorological answer text based on the atmospheric visibility value; and constructing a second question-and-answer text set based on all second meteorological question texts and all second meteorological answer texts.

[0107] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the atmospheric visibility detection method based on a multimodal model: constructing a first preset encoder, wherein the first preset encoder is used to extract features from each first historical image data to obtain first image features, and multiple fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded in a first preset position in the first meteorological question text corresponding to the first question text set; constructing a time series network structure, wherein the time series network structure is used to extract features from each first historical meteorological data to obtain first target time series features, and the first target time series features are embedded in a second preset position in the first meteorological question text; constructing a first initial multimodal model based on the first preset encoder, the time series network structure, and the preset multimodal model; constructing a first preset feature vector, and embedding the first preset feature vector in a third preset position in the first meteorological question text, and embedding the first preset feature vector in a fourth preset position in the first meteorological question text to obtain a first target question text set; and training the first initial multimodal model using the first target question text set based on a preset training strategy to obtain the first multimodal model.

[0108] Optionally, the aforementioned computer terminal can execute the program code for the following steps in the atmospheric visibility detection method based on a multimodal model: constructing a second preset encoder, wherein the second preset encoder is used to extract features from each second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded in the first preset position of the corresponding second meteorological question text in the second question text set; constructing a feature extraction network, wherein the feature extraction network is used to extract features from each second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded in the second preset position of the second meteorological question text; constructing a second initial multimodal model based on the second preset encoder, the feature extraction network, and the preset multimodal model; constructing a second preset feature vector, and embedding the second preset feature vector in the fifth preset position of the second meteorological question text, and embedding the second preset feature vector in the sixth preset position of the second meteorological question text to obtain the second target question text set; and training the second initial multimodal model using the second target question text set based on a preset training strategy to obtain the second multimodal model.

[0109] Optionally, Figure 9 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 9 As shown, the electronic device may include: one or more ( Figure 9 (Only one is shown) processor 902, memory 904, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0110] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the atmospheric visibility detection method and device based on a multimodal model in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned atmospheric visibility detection method based on a multimodal model. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0111] The processor can access the information and application programs stored in the memory via the transmission device to execute the steps described above in the atmospheric visibility detection method based on the multimodal model.

[0112] This application provides a scheme for atmospheric visibility detection based on a multimodal model. By constructing a text set of question-answer pairs and training a large multimodal model based on the text set, the deep features of image data and meteorological data such as temperature, humidity, air pressure, and wind speed can be transformed into inputs that the model can understand, thus achieving accurate prediction of atmospheric visibility. Furthermore, by adding preset feature vectors (such as learnable prompts) before and after the questions, the model's contextual understanding and complex logical reasoning capabilities can be enhanced, enabling it to maintain robustness and high accuracy in the face of diverse meteorological environments. This solves the technical problem of low accuracy in atmospheric visibility detection in related technologies.

[0113] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only. Electronic devices can also be terminal devices such as smartphones, tablets, PDAs, and mobile internet devices (MIDs). Figure 9 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 9 The different configurations shown.

[0114] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0115] Example 4

[0116] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the atmospheric visibility detection method based on a multimodal model provided in Embodiment 1.

[0117] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0118] This application also provides a computer program product, which, when executed on a data processing device, is suitable for performing steps of a method for detecting atmospheric visibility based on a multimodal model.

[0119] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0120] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0125] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / xxx> < / xxx> < / xxx> < / xxx> < / vis> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx> < / xxx>

Claims

1. A method for detecting atmospheric visibility based on a multimodal model, characterized in that, include: Acquire image data and meteorological data, wherein the image data is obtained through monitoring equipment deployed on the target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device; Based on the image data and the meteorological data, generate a preset question; The preset question is input into the target multimodal model, and the response information for the preset question is output. Based on the response information, the atmospheric visibility at the location of the target device is determined. The target multimodal model is one of a first multimodal model and a second multimodal model. The network structure of the first multimodal model includes at least a first preset encoder and a time series network structure. The network structure of the second multimodal model includes at least a second preset encoder and a feature extraction network. Before inputting the preset question into the target multimodal model and outputting the response information for the preset question, the method further includes: collecting second historical image data and second historical meteorological data from multiple devices at multiple times to obtain a data set; constructing a second question-to-text set based on the data set; and training a second initial multimodal model based on the second question-to-text set to obtain the second multimodal model. The step of constructing a second question-and-answer text set based on the data set includes: generating a second meteorological question text based on the temperature, humidity, air pressure, and wind speed in each of the second historical image data and each of the second historical meteorological data in the data set; generating a second meteorological answer text based on the atmospheric visibility value; and constructing a second question-and-answer text set based on all the second meteorological question texts and all the second meteorological answer texts. The steps of training a second initial multimodal model based on the second question-text set to obtain the second multimodal model include: constructing a second preset encoder, wherein the second preset encoder is used to extract features from each of the second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into a first preset position in the corresponding second meteorological question text in the second question-text set; constructing the feature extraction network, wherein the feature extraction network is used to extract features from each of the second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into the second meteorological question text. The second preset position; based on the second preset encoder, the feature extraction network, and the preset multimodal model, a second initial multimodal model is constructed; a second preset feature vector is constructed, and the second preset feature vector is embedded at the fifth preset position and the sixth preset position of the second meteorological question text to obtain a second target question text set; based on a preset training strategy, the second initial multimodal model is trained using the second target question text set to obtain the second multimodal model; the second preset feature vector is a learnable prompt word, the fifth preset position is the start position of the second meteorological question text, and the sixth preset position is the end position of the second meteorological question text.

2. The atmospheric visibility detection method based on a multimodal model according to claim 1, characterized in that, Before inputting the preset question into the target multimodal model and outputting the response information for the preset question, the method further includes: Collect first historical video data and first historical meteorological data from multiple devices within a preset time period, and extract frames from the first historical video data to obtain multiple first historical image data. Based on all the first historical meteorological data and all the first historical image data, a sequence data set is determined, wherein the first historical meteorological data includes at least: temperature, humidity, air pressure, wind speed and atmospheric visibility values. Based on the sequence data set, construct the first question pair text set; Based on the first question and the text set, the first initial multimodal model is trained to obtain the first multimodal model.

3. The atmospheric visibility detection method based on a multimodal model according to claim 2, characterized in that, The steps for constructing the first question pair text set based on the sequence data set include: For each first historical image data and each first historical meteorological data in the sequence data set, a first meteorological question text is generated based on the first historical image data and the temperature, humidity, air pressure and wind speed; Based on the atmospheric visibility value, a first meteorological response text is generated, wherein the first meteorological response text is a reply to the first meteorological question text; Based on all the first meteorological question texts and all the first meteorological answer texts, construct the first question-and-answer text set.

4. The atmospheric visibility detection method based on a multimodal model according to claim 3, characterized in that, The steps for training a first initial multimodal model based on the first question set of texts to obtain the first multimodal model include: Construct the first preset encoder, wherein the first preset encoder is used to extract features from each of the first historical image data to obtain first image features, and multiple fully connected layers are used to process the first image features to obtain first target image features, and the first target image features are embedded in the first preset position of the first meteorological question text corresponding to the first question text set. Construct the time series network structure, wherein the time series network structure is used to extract features from each of the first historical meteorological data to obtain a first target time series feature, and the first target time series feature is embedded in a second preset position in the first meteorological question text; Based on the first preset encoder, the time series network structure, and the preset multimodal model, the first initial multimodal model is constructed; Construct a first preset feature vector, embed the first preset feature vector at a third preset position in the first meteorological question text, and embed the first preset feature vector at a fourth preset position in the first meteorological question text to obtain a first target question pair text set; Based on a preset training strategy, the first initial multimodal model is trained using the first target question text set to obtain the first multimodal model.

5. A device for detecting atmospheric visibility based on a multimodal model, characterized in that, include: An acquisition unit is used to acquire image data and meteorological data, wherein the image data is obtained through monitoring equipment deployed on the target device, and the meteorological data is obtained through preset meteorological equipment deployed on the target device; The generation unit is used to generate a preset question based on the image data and the meteorological data; A determining unit is configured to input the preset question into a target multimodal model, output response information to the preset question, and determine the atmospheric visibility at the location of the target device based on the response information. The target multimodal model is one of a first multimodal model and a second multimodal model. The network structure of the first multimodal model includes at least a first preset encoder and a time-series network structure, and the network structure of the second multimodal model includes at least a second preset encoder and a feature extraction network. The atmospheric visibility detection device based on a multimodal model further includes: a second acquisition module, used to acquire second historical image data and second historical meteorological data from multiple devices at multiple time points before inputting the preset question into the target multimodal model and outputting response information to the preset question, to obtain a data set; a second construction module, used to construct a second question-pair text set based on the data set; and a second training module, used to train a second initial multimodal model based on the second question-pair text set, to obtain the second multimodal model. The second construction module includes: a third generation submodule, used to generate second meteorological question text based on the temperature, humidity, air pressure, and wind speed in each of the second historical image data and each of the second historical meteorological data in the dataset; a fourth generation submodule, used to generate second meteorological answer text based on atmospheric visibility values; and a second construction submodule, used to construct a second question-and-answer text set based on all the second meteorological question texts and all the second meteorological answer texts. The second training module includes: a seventh construction submodule, used to construct the second preset encoder, wherein the second preset encoder is used to extract features from each of the second historical image data to obtain second image features, and multiple fully connected layers are used to process the second image features to obtain second target image features, and the second target image features are embedded into a first preset position in the corresponding second meteorological question text in the second question text set; an eighth construction submodule, used to construct the feature extraction network, wherein the feature extraction network is used to extract features from each of the second historical meteorological data to obtain second target meteorological features, and the second target meteorological features are embedded into a second preset position in the second meteorological question text; a ninth construction submodule, used to construct based on the second preset encoder... The system includes an encoder, the feature extraction network, and a preset multimodal model. A second initial multimodal model is constructed. A tenth construction submodule is used to construct a second preset feature vector and embed the second preset feature vector at a fifth preset position and a sixth preset position in the second meteorological question text to obtain a second target question text set. A second training submodule is used to train the second initial multimodal model using the second target question text set based on a preset training strategy to obtain the second multimodal model. The second preset feature vector is a learnable prompt word, the fifth preset position is the start position of the second meteorological question text, and the sixth preset position is the end position of the second meteorological question text.

6. A computer program product, characterized in that, The method includes a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the atmospheric visibility detection method based on a multimodal model as described in any one of claims 1 to 4.

7. An electronic device, characterized in that, The device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the atmospheric visibility detection method based on a multimodal model as described in any one of claims 1 to 4.

Images