Rainfall prediction method and related device

Abstract

The invention discloses a rainfall prediction method and a related device, and relates to the field of artificial intelligence. Dividing the rainfall video into a plurality of shots through a shot segmentation technology; acquiring key frame sets respectively corresponding to the plurality of shots; obtaining a first sorting result, a second sorting result, a third sorting result and a fourth sorting result based on the plurality of key frame sets; inputting the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into a pre-constructed rainfall prediction model; and outputting the predicted rainfall through the rainfall prediction model. Therefore, accurate predicted rainfall can be obtained through the rainfall prediction model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a rainfall prediction method and related devices. Background Art

[0002] Among the weather change factors, rainfall is one of the important physical processes that drives the hydrological cycle of the basin. Its spatial distribution profoundly affects the spatial pattern of related variables such as surface runoff, floods, and soil moisture content, and drives the spatiotemporal changes in the amount of water resources in the basin. Therefore, timely and accurate estimation of rainfall is of great significance to the national economy, social life, and the safety of people's lives and property. Summary of the invention

[0003] In view of the above problems, this application provides a rainfall prediction method and related devices to achieve the purpose of accurately estimating rainfall. The specific scheme is as follows:

[0004] The first aspect of the present application provides a rainfall prediction method, comprising:

[0005] The rainfall video is divided into multiple shots through shot segmentation technology;

[0006] Acquire a plurality of key frame sets corresponding to the shots respectively, wherein the key frame sets corresponding to the shots include a first frame video image and a last frame video image of the shots;

[0007] For each of the video images, determining a first number of raindrop regions in the video image, each of the raindrop regions corresponding to a raindrop;

[0008] For each of the video images, based on the actual physical area corresponding to the video image and the first number, calculate the raindrop density corresponding to the video image;

[0009] For each raindrop area in each of the video images, based on the number of pixels included in the raindrop area, calculate the actual raindrop size of the raindrop area;

[0010] For each of the key frame sets, based on the time difference between the video images included in the key frame set and the displacement difference of the raindrop regions having the same characteristic information in the video images included in the key frame set, the raindrop velocity corresponding to the key frame set is calculated;

[0011] sorting the first numbers corresponding to the video images according to positions of the video images in the rainfall video to obtain a first sorting result;

[0012] sorting the raindrop densities corresponding to the video images according to the positions of the video images in the rainfall video to obtain a second sorting result;

[0013] sorting the raindrop speeds corresponding to the key frame set according to the positions of the key frame set in the rainfall video to obtain a third sorting result;

[0014] sorting actual raindrop sizes of the raindrop areas contained in the video images according to the positions of the video images in the rainfall video, so as to obtain a fourth sorting result;

[0015] Inputting the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into a pre-constructed rainfall prediction model; outputting predicted rainfall through the rainfall prediction model;

[0016] Among them, the rainfall prediction model takes the first sorting result corresponding to the sample rainfall video, the second sorting result corresponding to the sample rainfall video, the third sorting result corresponding to the rainfall video and the fourth sorting result corresponding to the sample rainfall video as input, and takes the actual rainfall corresponding to the sample rainfall video as the training target for training.

[0017] In a possible implementation, the step of determining a first number of raindrop areas in the video image includes:

[0018] Convert the video image in RGB color space into a first image in HSV color space;

[0019] Acquire a color histogram of the first image, where the color histogram includes a proportion of pixel values ​​corresponding to each pixel in the first image;

[0020] Based on the color histogram, determining a color threshold interval;

[0021] Determining that pixels in the first image whose pixel values ​​belong to the same color threshold interval belong to the same color region;

[0022] Setting the pixel values ​​of pixels belonging to the same color region in the first image to the same target pixel value to obtain a second image; the target pixel value corresponding to each color region belongs to the color threshold interval corresponding to the color region;

[0023] Acquire texture features from the second image to obtain a third image;

[0024] Processing the third image by an edge detection algorithm to obtain a fourth image;

[0025] performing binarization processing on the fourth image to obtain a fifth image;

[0026] A connected area consisting of pixels with a pixel value of 255 in the fifth image is determined to be the raindrop area, so as to obtain the first number of the raindrop areas.

[0027] In a possible implementation, the step of calculating the actual raindrop size of the raindrop area based on the number of pixels included in the raindrop area includes:

[0028] By formula , calculate the actual raindrop size W of the raindrop area O ,W p refers to the number of pixels contained in the raindrop area, W S It refers to the actual width of the camera that collects the rainfall video. sensor refers to the pixel width of the camera, f refers to the focal length of the camera, and d refers to the actual distance between the raindrops corresponding to the raindrop area and the camera.

[0029] In a possible implementation, the step of determining that the connected area formed by pixels with a pixel value of 255 in the fifth image is the raindrop area includes:

[0030] Traverse each pixel with a pixel value of 255 in the fifth image from left to right and from top to bottom, and if a first pixel above the pixel and a second pixel on the left side of the pixel are not assigned a label, assign a new label to the pixel;

[0031] Traversing each pixel whose pixel value is 255 in the fifth image from left to right and from top to bottom, if the first pixel and the second pixel corresponding to the pixel have both been assigned labels, determining the label of the pixel to be the minimum value of the labels corresponding to the first pixel and the second pixel;

[0032] For each pixel whose pixel value is 255 in the fifth image, if the label corresponding to the pixel is different from the label of the adjacent pixel of the pixel, it is determined that the label of the pixel and the label of the adjacent pixel have an equivalent relationship, and the adjacent pixels of the pixel include the pixel on the left side of the pixel, the pixel on the right side of the pixel, the pixel above the pixel, and the pixel below the pixel; the pixel value of the adjacent pixel is 255;

[0033] It is determined that the pixels in the fifth image having the equivalent relationship are located in the same connected area, so as to obtain a plurality of the raindrop areas in the fifth image.

[0034] In a possible implementation, the step of calculating the raindrop density corresponding to the video image based on the actual physical area corresponding to the video image and the first number includes:

[0035] By formula: , calculate the actual physical area A physical ; A image refers to the area of ​​the video image, W refers to the width of the video image, H refers to the height of the video image, S X It refers to the horizontal size of the camera that collects the rainfall video. y refers to the size of the camera in the vertical direction;

[0036] By formula: , the raindrop density D is calculated, and N refers to the first number.

[0037] In a possible implementation, the step of calculating the raindrop speed corresponding to the key frame set based on the time difference between the video images included in the key frame set and the displacement difference of the raindrop regions having the same characteristic information in the video images included in the key frame set includes:

[0038] For each of the video images included in the key frame set, acquiring feature information of each of the raindrop regions in the video image;

[0039] Acquire multiple groups of raindrop sets from the key frame set, each group of raindrop sets includes multiple raindrop regions whose feature information similarity is higher than or equal to a preset threshold, and the multiple raindrop regions included in each group of raindrop sets are located in different video images included in the key frame set;

[0040] For each group of raindrop sets, based on the time corresponding to the video images to which the multiple raindrop areas included in the raindrop set respectively belong and the positions of the multiple raindrop areas included in the raindrop set respectively located on the video images to which they belong, a raindrop speed corresponding to the raindrop set is calculated;

[0041] The raindrop speed corresponding to the key frame set is calculated based on the raindrop speeds respectively corresponding to the plurality of raindrop sets.

[0042] In a possible implementation, the step of acquiring feature information of each raindrop area in the video image includes:

[0043] Obtain the grayscale value of each feature point of the raindrop area in the video image.

[0044] In a possible implementation, the key frame set includes a first video image at time t of the rainfall video and a second video image at time t+Δt of the rainfall video, where Δt is greater than 0, and the step of obtaining multiple groups of raindrop sets from the key frame set includes:

[0045] It is determined that the raindrop regions corresponding to the feature points with the same grayscale value in the first video image and the second video image belong to the same raindrop set.

[0046] In a possible implementation, the feature point corresponding to the raindrop region belonging to the first video image and included in the raindrop set is a first feature point; the feature point corresponding to the raindrop region belonging to the second video image and included in the raindrop set is a second feature point;

[0047] The step of calculating the raindrop speed corresponding to the raindrop set based on the time corresponding to the video images to which the multiple raindrop areas included in the raindrop set respectively belong and the positions of the multiple raindrop areas included in the raindrop set respectively located on the video images to which they belong comprises:

[0048] Acquire a first actual physical size of the first video image;

[0049] Acquire a second actual physical size of the second video image;

[0050] Based on the first actual physical size and the relative position of the first feature point in the first video image, determine a first actual position (x A ,y A );

[0051] Based on the second actual physical size and the relative position of the second feature point in the second video image, determine a second actual position (x B ,y B );

[0052] By the formula △r=(x B -x A ,y B – y A ), calculate the displacement difference between the second actual position and the first actual position;

[0053] The raindrop speed corresponding to the raindrop set is calculated by the formula △r / △t, where △t is the difference between the moment when the second video image is located in the rainfall video and the moment when the first video image is located in the rainfall video.

[0054] A second aspect of the present application provides a rainfall prediction device, comprising:

[0055] The first division module is used to divide the rainfall video into multiple shots by using a shot segmentation technology;

[0056] A first acquisition module is used to acquire a set of key frames corresponding to each of the plurality of shots, wherein the set of key frames corresponding to the shots includes a first frame of video image and a last frame of video image of the shots;

[0057] A first determining module, configured to determine, for each of the video images, a first number of raindrop regions in the video image, each of the raindrop regions corresponding to one raindrop;

[0058] A first calculation module, configured to calculate, for each of the video images, a raindrop density corresponding to the video image based on an actual physical area corresponding to the video image and the first number;

[0059] A second calculation module is used to calculate, for each raindrop area in each of the video images, an actual raindrop size of the raindrop area based on the number of pixels included in the raindrop area;

[0060] A third calculation module is used to calculate, for each key frame set, a raindrop velocity corresponding to the key frame set based on a time difference between video images included in the key frame set and a displacement difference of raindrop regions having the same characteristic information in the video images included in the key frame set;

[0061] A first sorting module, configured to sort the first numbers corresponding to the video images according to positions of the video images in the rainfall video, so as to obtain a first sorting result;

[0062] A second sorting module, configured to sort the raindrop densities corresponding to the video images according to the positions of the video images in the rainfall video, so as to obtain a second sorting result;

[0063] A third sorting module, configured to sort the raindrop speeds corresponding to the key frame set according to the positions of the key frame set in the rainfall video, so as to obtain a third sorting result;

[0064] a fourth sorting module, configured to sort actual raindrop sizes of the raindrop areas contained in the video image according to the positions of the video images in the rainfall video, so as to obtain a fourth sorting result;

[0065] A second acquisition module is used to input the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into a pre-built rainfall prediction model; and output the predicted rainfall through the rainfall prediction model;

[0066] Among them, the rainfall prediction model takes the first sorting result corresponding to the sample rainfall video, the second sorting result corresponding to the sample rainfall video, the third sorting result corresponding to the rainfall video and the fourth sorting result corresponding to the sample rainfall video as input, and takes the actual rainfall corresponding to the sample rainfall video as the training target for training.

[0067] By means of the above technical scheme, the present application provides a rainfall prediction method, which can collect rainfall videos during rainfall; divide the rainfall videos into multiple shots by shot segmentation technology; obtain key frame sets corresponding to the multiple shots; for each video image, determine the first number of raindrop areas in the video image; for each video image, calculate the raindrop density corresponding to the video image based on the actual physical area corresponding to the video image and the first number; for each raindrop area in each video image, calculate the actual raindrop size of the raindrop area based on the number of pixels contained in the raindrop area; for each key frame set, based on the time difference between the video images contained in the key frame set and the displacement difference of the raindrop areas with the same characteristic information in the video images contained in the key frame set, Calculate the raindrop speed corresponding to the key frame set; sort the first number corresponding to the video image according to the position of the video image in the rainfall video to obtain a first sorting result; sort the raindrop density corresponding to the video image according to the position of the video image in the rainfall video to obtain a second sorting result; sort the raindrop speed corresponding to the key frame set according to the position of the key frame set in the rainfall video to obtain a third sorting result; sort the actual raindrop size of the raindrop area contained in the video image according to the position of the video image in the rainfall video to obtain a fourth sorting result; input the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into the pre-built rainfall prediction model; output the predicted rainfall through the rainfall prediction model. The first sorting result considers whether the raindrops are increasing, decreasing or unchanged as time goes by; the second sorting result considers whether the raindrop density is increasing, decreasing or unchanged as time goes by; the third sorting result considers whether the raindrop speed is increasing, decreasing or unchanged as time goes by. The fourth sorting result takes into account whether the raindrop size is getting bigger, smaller, or unchanged over time. Therefore, a relatively accurate rainfall prediction model can be used to obtain rainfall prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0069] Figure 1 A schematic diagram of a system architecture provided for this application;

[0070] Figure 2 A schematic diagram of an optional hardware structure of an observation terminal 100 provided in this application;

[0071] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0072] Figure 4 A flowchart of a rainfall prediction method provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram of an implementation method of the fifth image provided in an embodiment of the present application;

[0074] Figure 6 A schematic diagram of the structure of a rainfall prediction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0076] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0077] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0078] It is understandable that during the rainfall process, the rainfall amount can be predicted so as to interfere with the rainfall process. For example, if the rainfall amount does not reach the preset rainfall amount, the acoustic rainfall enhancement technology can be combined for rainfall enhancement; if the rainfall amount exceeds the preset rainfall amount, the clouds can be dispersed to reduce the rainfall. The following is an explanation of the rainfall prediction process.

[0079] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include an observation terminal 100, a server 200 and a collection terminal 300. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more collection terminals.

[0080] Among them, the collection terminal 300 may include one or more cameras, which are used to collect rainfall videos and send the above rainfall videos or video summaries of rainfall videos to the server 200; the server 200 obtains the predicted rainfall through the rainfall videos or the video summaries of rainfall videos, and the user can view the predicted rainfall through the observation terminal 100.

[0081] Exemplarily, the collection terminal 300 also includes a photosensitive sensor, a temperature sensor, a humidity sensor, an air pressure sensor, etc., so that the brightness changes, temperature changes, humidity changes, air pressure changes, etc. of the current environment can be recorded in real time during the process of the camera collecting and shooting the surrounding rainfall video.

[0082] Exemplarily, the collection terminals 300 may be respectively arranged at different locations.

[0083] Next describe Figure 1 The product form of the observation terminal 100;

[0084] The observation terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0085] Figure 2 An optional hardware structure diagram of the observation terminal 100 is shown.

[0086] refer to Figure 2As shown, the observation terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0087] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the observation terminal. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operation on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the observation terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0088] Among them, the input device 132 can receive input data and the like.

[0089] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the observation terminal 100, interactive interfaces, file displays and / or playback of any multimedia files. In the embodiment of the present application, the display unit 140 can be used to display an interface for displaying the predicted rainfall, etc.

[0090] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0091] The processor 170 is the control center of the observation terminal 100. It uses various interfaces and lines to connect various parts of the entire observation terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the observation terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0092] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0093] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0094] The observation terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0095] The observation terminal 100 also includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the observation terminal 100 to communicate with other devices, or to connect a charger to charge the observation terminal 100 .

[0096] Although not shown, the observation terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here. Figure 2 In the observation terminal 100 shown.

[0097] Next describe Figure 1 The product form of the server 200;

[0098] Figure 3 A schematic diagram of the structure of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0099] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0100] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0101] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0102] The memory 204 may be used to store software codes related to the rainfall prediction method, and the processor 202 may execute the steps of the rainfall prediction method of the chip, and may also schedule other units to implement corresponding functions.

[0103] It should be understood that the above-mentioned observation terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned observation terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0104] See also Figure 4 , is a flowchart of a rainfall prediction method provided in an embodiment of the present application, the method comprising the following steps S401 to S411.

[0105] Step S401: Divide the rainfall video into multiple shots using shot segmentation technology.

[0106] Exemplarily, the shot segmentation technology may be any one of a pixel-based shot segmentation technology, a histogram-based shot segmentation technology, an X2 histogram-based shot segmentation technology, and an edge profile change rate-based shot segmentation technology.

[0107] The shot in the embodiment of the present application refers to a video image sequence including a plurality of temporally continuous video images.

[0108] It is understandable that the rainfall video may be captured by multiple cameras. Different cameras have different acquisition directions, so there are differences between the captured video images. The video image sequences captured by different cameras correspond to different shots. For example, if the acquisition direction of the same camera can be changed, the video image sequences captured by the camera correspond to different shots before and after the camera's acquisition direction is changed. The rainfall video is obtained by splicing multiple shots together, so the boundaries of the shots need to be detected. After the boundaries of the shots are detected, multiple shots can be obtained.

[0109] For example, the shot boundary detection of the rainfall video may be performed using an SBD (Shot Boundary Detection) algorithm to obtain multiple shots.

[0110] Step S402: obtaining a plurality of key frame sets corresponding to the shots respectively, wherein the key frame sets corresponding to the shots include a first frame video image and a last frame video image of the shots.

[0111] It is understandable that the first frame of the video image of the shot represents the beginning of the shot, and the first frame of the video image usually includes the initial state of the background and the main observation object (which may be raindrops in this application). The last frame of the video image of the shot represents the end of the shot, and the last frame of the video image usually includes the final result of the main observation object (for example, whether the raindrops become larger or smaller).

[0112] Exemplarily, the key frame set of a shot may also include intermediate frame video images located at intermediate moments of the shot.

[0113] Exemplarily, the video image included in the key frame set can be denoised. Exemplarily, the video image can be denoised by preprocessing methods such as grayscale conversion and histogram equalization; Exemplarily, the video image can be denoised by denoising algorithms such as Gaussian filtering, median filtering or bilateral filtering.

[0114] Exemplarily, the video images contained in all key frame sets are sorted according to the moments of the video images in the rainfall video to obtain a key frame sequence. The key frame sequence forms a new video, namely, a video summary, so that the video content can be quickly understood based on the video summary. In summary, in the process of generating a video summary, the key frame sets corresponding to multiple shots need to be combined together. If the sizes of the video images contained in these key frame sets are inconsistent, the overall visual effect of the video summary will be affected and appear messy. By adjusting the size of the video images, the size and proportion of the video images contained in the video summary can be ensured to be consistent, thereby improving the consistency of the video summary.

[0115] Step S403: for each of the video images, determine a first number of raindrop regions in the video image, each of the raindrop regions corresponding to one raindrop.

[0116] Step S404: for each of the video images, based on the actual physical area corresponding to the video image and the first number, calculate the raindrop density corresponding to the video image.

[0117] For example, by formula: , calculate the actual physical area A physical ; A image refers to the area of ​​the video image, W refers to the width of the video image, H refers to the height of the video image, S X It refers to the horizontal size of the camera that collects the rainfall video. yrefers to the size of the camera in the vertical direction;

[0118] By formula: , the raindrop density D is calculated, and N refers to the first number.

[0119] The following example illustrates the actual physical area: Assuming that a camera captures a video image of a 3m×3m spatial area, the actual physical area corresponding to the video image is 3m×3m.

[0120] Step S405: for each raindrop region in each of the video images, based on the number of pixels included in the raindrop region, calculate the actual raindrop size of the raindrop region.

[0121] For example, if a region appears as a small dot at a high-resolution scale, but still presents a relatively small circular feature at a low-resolution scale, then the region is a raindrop region.

[0122] For example, by formula , calculate the actual raindrop size W of the raindrop area O ,W p refers to the number of pixels contained in the raindrop area, W S It refers to the actual width of the camera that collects the rainfall video. sensor refers to the pixel width of the camera, f refers to the focal length of the camera, and d refers to the actual distance between the raindrops corresponding to the raindrop area and the camera.

[0123] It will be appreciated that the actual raindrop size refers to the size of raindrops in the real world.

[0124] Step S406: for each key frame set, based on the time difference between the video images included in the key frame set and the displacement difference of the raindrop regions with the same characteristic information in the video images included in the key frame set, calculate the raindrop velocity corresponding to the key frame set.

[0125] Step S407: sorting the first numbers corresponding to the video images according to the positions of the video images in the rainfall video to obtain a first sorting result.

[0126] Exemplarily, the first sorting result can be used to determine whether the number of raindrops is increasing, decreasing, or unchanged as time goes by.

[0127] Step S408: sorting the raindrop densities corresponding to the video images according to the positions of the video images in the rainfall video to obtain a second sorting result.

[0128] Exemplarily, the second sorting result can be used to determine whether the raindrop density is increasing, decreasing, or remaining unchanged as time passes.

[0129] Step S409: sorting the raindrop speeds corresponding to the key frame set according to the positions of the key frame set in the rainfall video to obtain a third sorting result.

[0130] Exemplarily, the third sorting result can be used to determine whether the speed of raindrops is getting faster, slower, or unchanged as time goes by.

[0131] Step S410: sorting the actual raindrop sizes of the raindrop areas contained in the video image according to the positions of the video images in the rainfall video to obtain a fourth sorting result.

[0132] Exemplarily, the fourth sorting result can be used to determine whether the raindrop size is getting bigger, smaller, or unchanged as time passes.

[0133] Step S411: input the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into a pre-constructed rainfall prediction model; and output the predicted rainfall through the rainfall prediction model.

[0134] Among them, the rainfall prediction model takes the first sorting result corresponding to the sample rainfall video, the second sorting result corresponding to the sample rainfall video, the third sorting result corresponding to the rainfall video and the fourth sorting result corresponding to the sample rainfall video as input, and takes the actual rainfall corresponding to the sample rainfall video as the training target for training.

[0135] Exemplarily, the process of obtaining the first sorting result, the second sorting result, the third sorting result and the fourth sorting result of the sample rainfall video is the same as the process of obtaining the first sorting result, the second sorting result, the third sorting result and the fourth sorting result of the rainfall video.

[0136] For example, the plurality of sample rainfall videos may be rainfall videos shot under different backgrounds. Since the rainfall prediction model can analyze the rainfall trend through the first sorting result, the second sorting result, the third sorting result, and the fourth sorting result, an accurate rainfall amount can be predicted.

[0137] Exemplarily, the changes in ambient humidity, ambient temperature, ambient pressure, and ambient wind speed may also be detected by sensors, and the input of the rainfall prediction model may also include changes in ambient humidity, ambient temperature, ambient pressure, and ambient wind speed.

[0138] Exemplarily, each video image also corresponds to latitude and longitude information and acquisition time information.

[0139] The embodiment of the present application provides a rainfall prediction method, which can collect rainfall videos during rainfall; divide the rainfall videos into multiple shots by shot segmentation technology; obtain key frame sets corresponding to the multiple shots; for each video image, determine the first number of raindrop areas in the video image; for each video image, calculate the raindrop density corresponding to the video image based on the actual physical area corresponding to the video image and the first number; for each raindrop area in each video image, calculate the actual raindrop size of the raindrop area based on the number of pixels contained in the raindrop area; for each key frame set, calculate the raindrop density based on the time difference between the video images contained in the key frame set and the displacement difference of the raindrop areas with the same feature information in the video images contained in the key frame set. to the raindrop speed corresponding to the key frame set; sort the first number corresponding to the video image according to the position of the video image in the rainfall video to obtain a first sorting result; sort the raindrop density corresponding to the video image according to the position of the video image in the rainfall video to obtain a second sorting result; sort the raindrop speed corresponding to the key frame set according to the position of the key frame set in the rainfall video to obtain a third sorting result; sort the actual raindrop size of the raindrop area contained in the video image according to the position of the video image in the rainfall video to obtain a fourth sorting result; input the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into the pre-built rainfall prediction model; output the predicted rainfall through the rainfall prediction model. The first sorting result considers whether the raindrops are increasing, decreasing or unchanged as time goes by; the second sorting result considers whether the raindrop density is increasing, decreasing or unchanged as time goes by; the third sorting result considers whether the raindrop speed is increasing, decreasing or unchanged as time goes by. The fourth sorting result takes into account whether the raindrop size is getting bigger, smaller, or unchanged over time. Therefore, a relatively accurate rainfall prediction model can be used to obtain rainfall prediction.

[0140] In an optional implementation, there are multiple implementations of step S403, and the embodiments of the present application provide but are not limited to the following two.

[0141] The first implementation method of step S403 includes the following steps A11.

[0142] Step A11: input the video image into a pre-constructed raindrop number prediction model, and obtain a first number of raindrops contained in the video image through the raindrop number prediction model.

[0143] The raindrop number prediction model is obtained by taking the sample video image as input and the actual number of raindrops corresponding to the sample video image as the training target.

[0144] Exemplarily, the raindrop number prediction model may be one or more of a machine learning model, a deep learning model, and a large language model.

[0145] The second implementation of step S403 includes the following steps A21 to A29.

[0146] Step A21: Convert the video image in the RGB color space into a first image in the HSV color space.

[0147] The pixel value of each pixel in the video image in the RGB color space is mapped one by one to the HSV color space, thereby obtaining a first image. Exemplarily, the first image may contain up to 36 colors.

[0148] Step A22: Obtain a color histogram of the first image, where the color histogram includes the proportion of pixel values ​​corresponding to each pixel in the first image.

[0149] The color histogram describes the proportion of different colors in the entire video image, and does not care about the spatial position of each color.

[0150] Step A23: Determine a color threshold interval based on the color histogram.

[0151] After performing histogram analysis on the first image to obtain a color histogram, the main color distribution in the first image can be found.

[0152] Exemplarily, the color threshold interval is obtained manually through a color histogram based on experience.

[0153] It can be understood that since the focus is on raindrops in the video image, during the rainfall process, the main colors in the first image include the color of the raindrops; in order to avoid distinguishing raindrops from other objects, the color threshold interval of the raindrop color needs to be strictly set.

[0154] Exemplarily, the peaks in the color histogram generally indicate the intensity or frequency of a specific color in the first image, while the valleys provide contrast and variation information between colors. The peaks in the color histogram indicate that specific colors appear more frequently in the first image, and these colors may be the main colors in the first image or the color range of the target object (such as raindrops). By analyzing the peaks, the main hues and dominant colors of the first image can be determined. Valleys are opposite to peaks, and valleys represent transition areas or parts with low contrast between colors in the color histogram. The presence of valleys may indicate the contrast and separation between different colors in the first image, which is particularly useful for analyzing the background and foreground color distribution of the first image. Based on this, a color threshold area can be obtained based on the peak and valuation of the color histogram, for example, the valley is used as the maximum or minimum value of the color threshold interval.

[0155] Step A24: Determine whether pixels in the first image whose pixel values ​​belong to the same color threshold interval belong to the same color region.

[0156] Step A25: setting the pixel values ​​of pixels in the first image belonging to the same color area to the same target pixel value to obtain a second image; the target pixel value corresponding to each color area belongs to the color threshold interval corresponding to the color area.

[0157] Through steps A24 and A25, the first image is segmented according to different color threshold intervals to obtain a second image.

[0158] Step A26: Acquire texture features from the second image to obtain a third image.

[0159] Exemplarily, the second image may be processed by a filter or a LBP (Local Binary Pattern) histogram to obtain the third image, and the filter is used to highlight texture features in the second image.

[0160] Exemplarily, the filter or LBP histogram may extract features of the first image, such as mean and variance, to distinguish different types of textures.

[0161] Step A27: Process the third image using an edge detection algorithm to obtain a fourth image.

[0162] Exemplarily, edge detection is performed on the third image to identify points with obvious brightness changes in the third image, such as the boundary of a raindrop region in the third image, where the raindrop region refers to the outline of a raindrop. Exemplarily, the edge detection algorithm may be a Canny edge algorithm.

[0163] It can be understood that, since the third image highlights the texture features, the outline of the raindrop, ie, the boundary, can be accurately obtained through the edge detection algorithm.

[0164] Exemplarily, the outline or shape feature of the raindrops is obtained through A27.

[0165] The color features, texture features and shape features of the video image are obtained through the above steps A22 to A27, so that the background and foreground (ie, raindrops) of the video image can be distinguished.

[0166] Step A28: binarize the fourth image to obtain a fifth image.

[0167] It is understandable that the fourth image after binarization is referred to as the fifth image in the present application, and the pixel color of the pixels in the fifth image is black or white, wherein the raindrops are white and the background is black.

[0168] Exemplarily, during the binarization process of the fourth image, a grayscale threshold may be set. If the pixel value of a pixel is greater than the grayscale threshold, the pixel value of the pixel is set to 255; otherwise, the pixel value of the pixel is set to 0.

[0169] Step A29: Determine that the connected area formed by pixels with a pixel value of 255 in the fifth image is the raindrop area, so as to obtain the first number of the raindrop areas.

[0170] Exemplarily, the fifth image includes a plurality of raindrop regions.

[0171] In order for those skilled in the art to better understand the fifth image mentioned in the embodiment of the present application, the fifth image is described below by way of example. Figure 5 Shown is a schematic diagram of an implementation method of the fifth image provided in an embodiment of the present application.

[0172] Figure 5 In the figure, the white area is the raindrop area and the black area is the background area.

[0173] Exemplarily, the amount of rainfall may be determined by the number of raindrop regions included in the fifth image.

[0174] Combination Figure 5 It can be seen that overlapping raindrops or adjacent raindrops can be regarded as the same raindrop. Figure 5 It can be seen that after the processing from step A21 to step A28, a series of feature information has been extracted, such as raindrop size, raindrop shape, etc.

[0175] In an optional implementation, there are multiple implementations of step A29, and the embodiments of the present application provide but are not limited to the following two methods.

[0176] The first method for implementing step A29 includes but is not limited to the following steps B11 to B12.

[0177] Step B11: If the connected area is a multiply connected area, and the ratio of the number of pixels contained in the multiply connected area to the number of pixels contained in the outer contour of the multiply connected area is higher than or equal to a preset ratio threshold, the pixel value of the pixel with a pixel value of 0 in the outer contour of the multiply connected area is set to 255.

[0178] Exemplarily, the connected region may be a simply connected region or a multiply connected region.

[0179] Step B12: Calculate the first number of simply connected regions consisting of pixels with a pixel value of 255 in the fifth image.

[0180] In order to facilitate those skilled in the art to better understand the present method, the following examples are given to illustrate. Figure 5 It can be seen that the outer contour of the multi-connected region 501 contains a circular black area. This may be caused by external ambient light, that is, the part originally belonging to the raindrops is recognized as the background area. This problem can be solved by the above steps B11 to B12.

[0181] The second method for implementing step A29 includes but is not limited to the following steps B21 to B24.

[0182] Step B21: Traverse each pixel with a pixel value of 255 in the fifth image from left to right and from top to bottom. If the first pixel above the pixel and the second pixel on the left of the pixel are not assigned a label, assign a new label to the pixel.

[0183] Exemplarily, during the traversal process, no operation is performed on pixels with a pixel value of 0, that is, no label is assigned.

[0184] It can be understood that since the fifth image is traversed from left to right and from top to bottom, if the first pixel above the pixel and the second pixel to the left of the pixel are not assigned a label, it means that the pixel values ​​of the first pixel and the second pixel are 0, which is the boundary of the connected area.

[0185] Exemplarily, a label counter may be set. The initial value of a label generated by the label counter is 1, and the label value is accumulated in sequence. For example, the label value of the tenth label is 10.

[0186] Step B22: Traverse each pixel with a pixel value of 255 in the fifth image from left to right and from top to bottom. If the first pixel and the second pixel corresponding to the pixel have been assigned labels, determine that the label of the pixel is the minimum value of the labels corresponding to the first pixel and the second pixel.

[0187] It can be understood that, since the fifth image is traversed from left to right and from top to bottom, if the first pixel above the pixel and the second pixel on the left of the pixel have both been assigned labels, it means that the pixel values ​​of the first pixel and the second pixel are 255, which is inside the connected area. Since the pixel is adjacent to the first pixel and the second pixel, they should belong to the same raindrop, so the label of the pixel is determined to be the minimum value of the labels corresponding to the first pixel and the second pixel.

[0188] Step B23: For each pixel with a pixel value of 255 in the fifth image, if the label corresponding to the pixel is different from the label of the adjacent pixels of the pixel, determine that the label of the pixel and the label of the adjacent pixels have an equivalent relationship, and the adjacent pixels of the pixel include pixels to the left of the pixel, pixels to the right of the pixel, pixels above the pixel, and pixels below the pixel; the pixel value of the adjacent pixels is 255.

[0189] Exemplarily, the equivalence relationship can be recorded in an equivalence relationship table. For example, if the equivalence relationship table includes the correspondence between a label with a label value of 5, a label with a label value of 6, a label with a label value of 7, and a label with a label value of 8, it means that the label with a label value of 5, the label with a label value of 6, the label with a label value of 7, and the label with a label value of 8 have an equivalence relationship.

[0190] Exemplarily, step B22 and step B23 may be repeated, and if the equivalence relationship changes, the equivalence relationship table is updated.

[0191] Step B24: Determine whether the pixels in the fifth image having the equivalent relationship are located in the same connected region, so as to obtain a plurality of the raindrop regions in the fifth image.

[0192] In an optional implementation, there are multiple implementations of step S406, and the present application embodiment provides but is not limited to the following method. The method includes the following steps C11 to C14.

[0193] Step C11: for each of the video images included in the key frame set, obtain feature information of each of the raindrop regions in the video image.

[0194] Exemplarily, the feature information of the raindrop region may include the grayscale value of the feature point of the raindrop region. Exemplarily, the feature point includes but is not limited to: corner point and edge point. Corner point refers to a point with a direction that has a significant change in the fifth image, usually located at the intersection of two edges in different directions, and edge point refers to a point with a strong gradient change in the fifth image, usually located at the boundary of an object such as a raindrop.

[0195] In the related art, rainfall is predicted by analyzing video images in rainfall videos. It is understandable that in natural environments, drastic changes in lighting conditions will seriously interfere with rainfall estimation. For example, under extremely strong direct sunlight, some areas in the video image may lose details, making some features in the video image blurred and affecting the recognition rate; at night or in low illumination conditions, the noise in the video image increases and the contrast decreases, which will also interfere with the accuracy of some feature extraction and reduce the accuracy of rainfall estimation. Corner points and edge points usually appear in unique positions in video images and have unique structures. Even if the video image is affected by rotation, scaling or lighting changes, these points can still be reliably detected in the video image, thereby solving the above problems.

[0196] Step C12: Acquire multiple groups of raindrop sets from the key frame set, each group of raindrop sets includes multiple raindrop areas whose feature information similarity is greater than or equal to a preset threshold, and the multiple raindrop areas contained in each group of raindrop sets are located in different video images contained in the key frame set.

[0197] Exemplarily, the preset threshold may be determined based on actual conditions and is not limited here.

[0198] It can be understood that a group of key frame sets corresponds to one shot, and the multiple raindrop areas contained in each group of raindrop sets are located in different video images contained in the same key frame set, and each group of raindrop sets includes multiple raindrop areas whose feature information similarity is higher than or equal to a preset threshold, which means that the multiple raindrop areas located in different video images whose feature information similarity is higher than or equal to the preset threshold should belong to the same raindrop, and the same raindrop area in different video images can be tracked to obtain the raindrop speed of the raindrop.

[0199] Exemplarily, step C12 may be performed by a KLT (Kanade-Lucas-Tomasi Tracking Method) tracking algorithm. The KLT algorithm mainly relies on the optical flow equation, assuming that the grayscale value of the same feature point remains unchanged between two consecutive video frames, that is, the grayscale value I(x, y, t) of the feature point p=(x, y) at time t is the same as the grayscale value I(x+Δx, y+Δy, t+ Δt) at the position moved by (Δx, Δy) at time t+ Δt, that is, I(x, y, t)= I(x+Δx, y+Δy, t+ Δt).

[0200] Where I represents the brightness function of the video image, (x, y) represents the position of the pixel, and t represents the time when the video image is located in the rainfall video.

[0201] In summary, the grayscale values ​​of the same feature points in each raindrop area in each group of raindrop sets are the same.

[0202] Exemplarily, assuming that a key frame set includes: a first video image located at the tth moment of the rainfall video and a second video image located at the t+△tth moment of the rainfall video, △t is greater than 0; step C12 specifically includes: determining that the raindrop areas corresponding to the feature points with the same grayscale values ​​in the first video image and the second video image belong to the same raindrop set.

[0203] Step C13: For each group of raindrop sets, based on the time corresponding to the video images to which the multiple raindrop areas included in the raindrop set respectively belong and the positions of the multiple raindrop areas included in the raindrop set on the video images to which they respectively belong, calculate the raindrop speed corresponding to the raindrop set.

[0204] Exemplarily, step C13 specifically includes the following steps D1 to D6.

[0205] Exemplarily, assuming that a key frame set includes: a first video image at the tth moment of the rainfall video and a second video image at the t+△tth moment of the rainfall video, △t being greater than 0; step C12 specifically includes: determining that the raindrop regions corresponding to the feature points with the same grayscale value in the first video image and the second video image belong to the same raindrop set. The feature points corresponding to the raindrop regions belonging to the first video image included in the raindrop set are first feature points; the feature points corresponding to the raindrop regions belonging to the second video image included in the raindrop set are second feature points.

[0206] Step D1: Acquire a first actual physical size of the first video image.

[0207] Step D2: Acquire a second actual physical size of the second video image.

[0208] The first actual physical size and the second actual physical size refer to sizes in the real world. For example, if a camera captures a rainfall video in a spatial area of ​​3×3 meters, the first actual physical size is 3×3.

[0209] Illustratively, the first actual physical size and the second actual physical size may be the same or different.

[0210] Step D3: Determine a first actual position (x ) of the first feature point based on the first actual physical size and the relative position of the first feature point in the first video image. A ,y A ).

[0211] Step D4: Determine a second actual position (x ) of the second feature point based on the second actual physical size and the relative position of the second feature point in the second video image. B ,y B ).

[0212] Step D5: Use the formula △r=(x B -x A ,y B – y A ), and calculate the displacement difference between the second actual position and the first actual position.

[0213] Step D6: Calculate the raindrop velocity corresponding to the raindrop set by the formula △r / △t, where △t is the difference between the moment when the second video image is located in the rainfall video and the moment when the first video image is located in the rainfall video.

[0214] Step C14: based on the raindrop velocities corresponding to the plurality of raindrop sets, respectively, the raindrop velocity corresponding to the key frame set is calculated.

[0215] Exemplarily, step C14 specifically includes: determining an average of raindrop velocities corresponding to a plurality of raindrop sets included in the key frame set, as the raindrop velocity corresponding to the key frame set.

[0216] Exemplarily, the formula for the raindrop velocity corresponding to the key frame set is as follows:

[0217] , where v x,h It refers to the speed in the x-axis direction corresponding to the h-th key frame set, v y,h It refers to the speed in the y-axis direction corresponding to the h-th key frame set, and J is the total number of key frame sets.

[0218] in, , refers to the raindrop rate; Refers to the direction of the raindrop. The angle is calculated by the arctan2 function, ensuring that the result is in the range [-Π, Π].

[0219] In an optional implementation, the rainfall prediction model may be a random forest regression model. The formula corresponding to the rainfall prediction model is as follows:

[0220] , where f g (u) represents the predicted rainfall of the sample rainfall video u by the g-th decision tree, G is the total number of decision trees contained in the random forest regression model; S represents the predicted rainfall output by the rainfall prediction model. u represents the first sorting result, the second sorting result, the third sorting result, and the fourth sorting result of the sample rainfall video.

[0221] Exemplarily, the predicted rainfall obtained by the rainfall prediction model and the actual rainfall in the sample rainfall video may be compared to obtain an error, and the prediction performance of the rainfall prediction model may be evaluated based on the error.

[0222] The following is an explanation of the process of training the rainfall prediction model. The process includes: randomly and evenly dividing multiple sample rainfall videos into R subsets of equal size, each subset includes multiple sample rainfall videos. Each subset includes representative sample rainfall videos to ensure the balance of the distribution of sample rainfall videos, and the subsets are repeatedly trained and verified, as follows:

[0223] The nth subset is selected as the validation set, and the remaining Rn subsets are combined as the training set. The rainfall prediction model is trained with the training set, and the trained rainfall prediction model is evaluated according to the validation set. After completing R iterations, the performance evaluation results of the R iterations are averaged. According to the average value of the performance evaluation results, the rainfall prediction model parameters are adjusted to further improve the performance of the rainfall prediction model.

[0224] Repeat the above steps and continuously iterate the training until the performance of the rainfall prediction model on the validation set reaches the optimal level. Then use the rainfall prediction model as the final model, predict the rainfall based on the final model, and transmit the measured rainfall to the observation terminal in real time.

[0225] The embodiments of the present application can also achieve high-precision raindrop recognition in environments with poor lighting or complex backgrounds, thereby enhancing the overall accuracy of rainfall detection.

[0226] A rainfall prediction method provided in an embodiment of the present application is introduced above, and a device for executing the above rainfall prediction method will be introduced below.

[0227] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a rainfall prediction device provided in an embodiment of the present application. Figure 6 As shown, the rainfall prediction device includes: a first division module 601, a first acquisition module 602, a first determination module 603, a first calculation module 604, a second calculation module 605, a third calculation module 606, a first sorting module 607, a second sorting module 608, a third sorting module 609, a fourth sorting module 610 and a second acquisition module 611, wherein:

[0228] The first division module 601 is used to divide the rainfall video into multiple shots by using a shot segmentation technology;

[0229] A first acquisition module 602 is used to acquire a plurality of key frame sets corresponding to the shots, wherein the key frame sets corresponding to the shots include a first frame video image and a last frame video image of the shot;

[0230] A first determining module 603 is used to determine, for each of the video images, a first number of raindrop regions in the video image, each of the raindrop regions corresponding to one raindrop;

[0231] A first calculation module 604 is used to calculate, for each of the video images, a raindrop density corresponding to the video image based on an actual physical area corresponding to the video image and the first number;

[0232] A second calculation module 605 is used to calculate, for each raindrop area in each of the video images, an actual raindrop size of the raindrop area based on the number of pixels included in the raindrop area;

[0233] A third calculation module 606 is used to calculate, for each key frame set, a raindrop velocity corresponding to the key frame set based on a time difference between video images included in the key frame set and a displacement difference of raindrop regions having the same characteristic information in the video images included in the key frame set;

[0234] A first sorting module 607, configured to sort the first numbers corresponding to the video images according to the positions of the video images in the rainfall video, so as to obtain a first sorting result;

[0235] A second sorting module 608 is used to sort the raindrop densities corresponding to the video images according to the positions of the video images in the rainfall video to obtain a second sorting result;

[0236] A third sorting module 609 is used to sort the raindrop speeds corresponding to the key frame set according to the positions of the key frame set in the rainfall video to obtain a third sorting result;

[0237] A fourth sorting module 610 is used to sort the actual raindrop sizes of the raindrop areas contained in the video image according to the positions of the video images in the rainfall video, so as to obtain a fourth sorting result;

[0238] The second acquisition module 611 is used to input the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into a pre-built rainfall prediction model; and output the predicted rainfall through the rainfall prediction model;

[0239] Among them, the rainfall prediction model takes the first sorting result corresponding to the sample rainfall video, the second sorting result corresponding to the sample rainfall video, the third sorting result corresponding to the rainfall video and the fourth sorting result corresponding to the sample rainfall video as input, and takes the actual rainfall corresponding to the sample rainfall video as the training target for training.

[0240] In an optional implementation, the first determining module includes:

[0241] A conversion unit, configured to convert the video image in the RGB color space into a first image in the HSV color space;

[0242] A first acquisition unit, configured to acquire a color histogram of the first image, wherein the color histogram includes a proportion of pixel values ​​corresponding to each pixel in the first image;

[0243] A first determining unit, configured to determine a color threshold interval based on the color histogram;

[0244] A second determining unit, configured to determine that pixels in the first image whose pixel values ​​belong to the same color threshold interval belong to the same color region;

[0245] A setting unit, used for setting the pixel values ​​of pixels belonging to the same color region in the first image to the same target pixel value, so as to obtain a second image; the target pixel value corresponding to each color region belongs to the color threshold interval corresponding to the color region;

[0246] A second acquisition unit, used for acquiring texture features from the second image to obtain a third image;

[0247] an edge detection unit, configured to process the third image by using an edge detection algorithm to obtain a fourth image;

[0248] A binarization unit, used for performing binarization processing on the fourth image to obtain a fifth image;

[0249] The third determining unit is used to determine that the connected area composed of pixels with a pixel value of 255 in the fifth image is the raindrop area, so as to obtain the first number of the raindrop areas.

[0250] In an optional implementation, the second calculation module is specifically used for:

[0251] By formula , calculate the actual raindrop size W of the raindrop area O ,W p refers to the number of pixels contained in the raindrop area, W S It refers to the actual width of the camera that collects the rainfall video. sensor refers to the pixel width of the camera, f refers to the focal length of the camera, and d refers to the actual distance between the raindrops corresponding to the raindrop area and the camera.

[0252] In an optional implementation, the third determining unit includes:

[0253] A first allocation subunit is configured to sequentially traverse each pixel with a pixel value of 255 in the fifth image from left to right and from top to bottom, and if a first pixel above the pixel and a second pixel on the left side of the pixel are not allocated a label, allocate a new label to the pixel;

[0254] A first determining subunit is configured to sequentially traverse each pixel having a pixel value of 255 in the fifth image from left to right and from top to bottom, and if the first pixel and the second pixel corresponding to the pixel have both been assigned labels, determine that the label of the pixel is the minimum value of the labels corresponding to the first pixel and the second pixel;

[0255] a second determining subunit, for each pixel having a pixel value of 255 in the fifth image, if a label corresponding to the pixel is different from a label of an adjacent pixel of the pixel, determining that the label of the pixel has an equivalent relationship with the label of the adjacent pixel, the adjacent pixels of the pixel including a pixel to the left of the pixel, a pixel to the right of the pixel, a pixel above the pixel, and a pixel below the pixel; the pixel value of the adjacent pixel is 255;

[0256] The third determining subunit is used to determine that the pixels having the equivalent relationship in the fifth image are located in the same connected area, so as to obtain the multiple raindrop areas in the fifth image.

[0257] In an optional implementation, the first calculation module is specifically used to: use the formula: , calculate the actual physical area A physical ; A image refers to the area of ​​the video image, W refers to the width of the video image, H refers to the height of the video image, S X It refers to the horizontal size of the camera that collects the rainfall video. y refers to the size of the camera in the vertical direction;

[0258] By formula: , the raindrop density D is calculated, and N refers to the first number.

[0259] In an optional implementation, the third calculation module includes:

[0260] A third acquisition unit, configured to acquire, for each of the video images included in the key frame set, feature information of each of the raindrop regions in the video image;

[0261] a fourth acquisition unit, configured to acquire multiple groups of raindrop sets from the key frame set, each group of raindrop sets including multiple raindrop regions whose feature information similarity is greater than or equal to a preset threshold, and the multiple raindrop regions included in each group of raindrop sets are located in different video images included in the key frame set;

[0262] A first calculation unit is configured to calculate, for each group of raindrop sets, a raindrop velocity corresponding to the raindrop set based on the time corresponding to the video images to which the multiple raindrop areas included in the raindrop set respectively belong and the positions of the multiple raindrop areas included in the raindrop set respectively located on the video images to which they belong;

[0263] The second calculation unit is used to calculate the raindrop speed corresponding to the key frame set based on the raindrop speeds respectively corresponding to the plurality of raindrop sets.

[0264] In an optional implementation, the third acquisition unit includes: an acquisition subunit, configured to acquire the grayscale value of each feature point of the raindrop area in the video image.

[0265] In an optional implementation, the key frame set includes a first video image at time t of the rainfall video and a second video image at time t+Δt of the rainfall video, where Δt is greater than 0, and the fourth acquisition unit includes:

[0266] The fourth determining subunit is used to determine that the raindrop regions corresponding to the feature points with the same grayscale value in the first video image and the second video image belong to the same raindrop set.

[0267] In an optional implementation, the feature points corresponding to the raindrop region belonging to the first video image included in the raindrop set are first feature points; the feature points corresponding to the raindrop region belonging to the second video image included in the raindrop set are second feature points; and the first calculation unit includes:

[0268] A first physical size acquisition subunit, used to acquire a first actual physical size of the first video image;

[0269] A second physical size acquisition subunit, used to acquire a second actual physical size of the second video image;

[0270] a fifth determining subunit, configured to determine a first actual position (x ) of the first feature point based on the first actual physical size and a relative position of the first feature point in the first video image; A ,y A );

[0271] a sixth determining subunit, configured to determine a second actual position (x ) of the second feature point based on the second actual physical size and the relative position of the second feature point in the second video image; B ,y B );

[0272] The first calculation subunit is used to calculate the B -x A ,y B – y A ), calculate the displacement difference between the second actual position and the first actual position;

[0273] The second calculation subunit is used to calculate the raindrop speed corresponding to the raindrop set through the formula △r / △t, where △t is the difference between the moment when the second video image is located in the rainfall video and the moment when the first video image is located in the rainfall video.

[0274] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on a server, the server implements any rainfall prediction method provided in the embodiment of the present application.

[0275] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by a server, the server can implement any rainfall prediction method provided in the embodiment of the present application.

[0276] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0277] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0278] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0279] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A rainfall prediction method, characterized in that: include: The rainfall video is divided into multiple shots through shot segmentation technology; Acquire a plurality of key frame sets corresponding to the shots respectively, wherein the key frame sets corresponding to the shots include a first frame video image and a last frame video image of the shots; For each of the video images, determining a first number of raindrop regions in the video image, each of the raindrop regions corresponding to a raindrop; For each of the video images, based on the actual physical area corresponding to the video image and the first number, calculate the raindrop density corresponding to the video image; For each raindrop area in each of the video images, based on the number of pixels included in the raindrop area, calculate the actual raindrop size of the raindrop area; For each of the key frame sets, based on the time difference between the video images included in the key frame set and the displacement difference of the raindrop regions having the same characteristic information in the video images included in the key frame set, the raindrop velocity corresponding to the key frame set is calculated; sorting the first numbers corresponding to the video images according to positions of the video images in the rainfall video to obtain a first sorting result; sorting the raindrop densities corresponding to the video images according to the positions of the video images in the rainfall video to obtain a second sorting result; sorting the raindrop speeds corresponding to the key frame set according to the positions of the key frame set in the rainfall video to obtain a third sorting result; sorting actual raindrop sizes of the raindrop areas contained in the video images according to the positions of the video images in the rainfall video, so as to obtain a fourth sorting result; Inputting the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into a pre-constructed rainfall prediction model; outputting predicted rainfall through the rainfall prediction model; Among them, the rainfall prediction model takes the first sorting result corresponding to the sample rainfall video, the second sorting result corresponding to the sample rainfall video, the third sorting result corresponding to the rainfall video and the fourth sorting result corresponding to the sample rainfall video as input, and takes the actual rainfall corresponding to the sample rainfall video as the training target for training.

2. The rainfall prediction method according to claim 1, characterized in that: The step of determining the first number of raindrop areas in the video image comprises: Convert the video image in RGB color space into a first image in HSV color space; Acquire a color histogram of the first image, where the color histogram includes a proportion of pixel values ​​corresponding to each pixel in the first image; Based on the color histogram, determining a color threshold interval; Determining that pixels in the first image whose pixel values ​​belong to the same color threshold interval belong to the same color region; Setting the pixel values ​​of pixels belonging to the same color region in the first image to the same target pixel value to obtain a second image; the target pixel value corresponding to each color region belongs to the color threshold interval corresponding to the color region; Acquire texture features from the second image to obtain a third image; Processing the third image by an edge detection algorithm to obtain a fourth image; performing binarization processing on the fourth image to obtain a fifth image; A connected area consisting of pixels with a pixel value of 255 in the fifth image is determined to be the raindrop area, so as to obtain the first number of the raindrop areas.

3. The rainfall prediction method according to claim 1 or 2, characterized in that: The step of calculating the actual raindrop size of the raindrop area based on the number of pixels contained in the raindrop area comprises: By formula , calculate the actual raindrop size W of the raindrop area O ,W p refers to the number of pixels contained in the raindrop area, W S It refers to the actual width of the camera that collects the rainfall video. sensor refers to the pixel width of the camera, f refers to the focal length of the camera, and d refers to the actual distance between the raindrops corresponding to the raindrop area and the camera.

4. The rainfall prediction method according to claim 2, characterized in that: The step of determining that the connected area formed by pixels with a pixel value of 255 in the fifth image is the raindrop area comprises: Traverse each pixel with a pixel value of 255 in the fifth image from left to right and from top to bottom, and if a first pixel above the pixel and a second pixel on the left side of the pixel are not assigned a label, assign a new label to the pixel; Traversing each pixel whose pixel value is 255 in the fifth image from left to right and from top to bottom, if the first pixel and the second pixel corresponding to the pixel have both been assigned labels, determining the label of the pixel to be the minimum value of the labels corresponding to the first pixel and the second pixel; For each pixel whose pixel value is 255 in the fifth image, if the label corresponding to the pixel is different from the label of the adjacent pixel of the pixel, it is determined that the label of the pixel and the label of the adjacent pixel have an equivalent relationship, and the adjacent pixels of the pixel include the pixel on the left side of the pixel, the pixel on the right side of the pixel, the pixel above the pixel, and the pixel below the pixel; the pixel value of the adjacent pixel is 255; It is determined that the pixels in the fifth image having the equivalent relationship are located in the same connected region, so as to obtain a plurality of the raindrop regions in the fifth image.

5. The rainfall prediction method according to claim 1, characterized in that: The step of calculating the raindrop density corresponding to the video image based on the actual physical area corresponding to the video image and the first number includes: By formula: , calculate the actual physical area A physical ; A image refers to the area of ​​the video image, W refers to the width of the video image, H refers to the height of the video image, S X It refers to the horizontal size of the camera that collects the rainfall video. y refers to the size of the camera in the vertical direction; By formula: , the raindrop density D is calculated, and N refers to the first number.

6. The rainfall prediction method according to claim 1, characterized in that: The step of calculating the raindrop speed corresponding to the key frame set based on the time difference between the video images included in the key frame set and the displacement difference of the raindrop regions having the same characteristic information in the video images included in the key frame set comprises: For each of the video images included in the key frame set, acquiring feature information of each of the raindrop regions in the video image; Acquire multiple groups of raindrop sets from the key frame set, each group of raindrop sets includes multiple raindrop regions whose feature information similarity is higher than or equal to a preset threshold, and the multiple raindrop regions included in each group of raindrop sets are located in different video images included in the key frame set; For each group of raindrop sets, based on the time corresponding to the video images to which the multiple raindrop areas included in the raindrop set respectively belong and the positions of the multiple raindrop areas included in the raindrop set respectively located on the video images to which they belong, a raindrop speed corresponding to the raindrop set is calculated; The raindrop speed corresponding to the key frame set is calculated based on the raindrop speeds respectively corresponding to the plurality of raindrop sets.

7. The rainfall prediction method according to claim 6, characterized in that: The step of obtaining characteristic information of each raindrop area in the video image comprises: Obtain the grayscale value of each feature point of the raindrop area in the video image.

8. The rainfall prediction method according to claim 7, characterized in that: The key frame set includes a first video image at time t of the rainfall video and a second video image at time t+Δt of the rainfall video, where Δt is greater than 0, and the step of obtaining multiple groups of raindrop sets from the key frame set includes: It is determined that the raindrop regions corresponding to the feature points with the same grayscale value in the first video image and the second video image belong to the same raindrop set.

9. The rainfall prediction method according to claim 8, characterized in that: The feature points corresponding to the raindrop region belonging to the first video image and included in the raindrop set are first feature points; the feature points corresponding to the raindrop region belonging to the second video image and included in the raindrop set are second feature points; The step of calculating the raindrop speed corresponding to the raindrop set based on the time corresponding to the video images to which the multiple raindrop areas included in the raindrop set respectively belong and the positions of the multiple raindrop areas included in the raindrop set respectively located on the video images to which they belong comprises: Acquire a first actual physical size of the first video image; Acquire a second actual physical size of the second video image; Based on the first actual physical size and the relative position of the first feature point in the first video image, determine a first actual position (x A ,y A ); Based on the second actual physical size and the relative position of the second feature point in the second video image, determine a second actual position (x B ,y B ); By the formula △r=(x B -x A ,y B – y A ), calculate the displacement difference between the second actual position and the first actual position; The raindrop speed corresponding to the raindrop set is calculated by the formula △r / △t, where △t is the difference between the moment when the second video image is located in the rainfall video and the moment when the first video image is located in the rainfall video.

10. A rainfall prediction device, characterized in that: include: The first division module is used to divide the rainfall video into multiple shots by using a shot segmentation technology; A first acquisition module is used to acquire a plurality of key frame sets corresponding to the shots, wherein the key frame sets corresponding to the shots include a first frame video image and a last frame video image of the shots; A first determining module, configured to determine, for each of the video images, a first number of raindrop regions in the video image, each of the raindrop regions corresponding to one raindrop; A first calculation module, configured to calculate, for each of the video images, a raindrop density corresponding to the video image based on an actual physical area corresponding to the video image and the first number; A second calculation module is used to calculate, for each raindrop area in each of the video images, an actual raindrop size of the raindrop area based on the number of pixels included in the raindrop area; A third calculation module is used to calculate, for each key frame set, a raindrop velocity corresponding to the key frame set based on a time difference between video images included in the key frame set and a displacement difference of raindrop regions having the same characteristic information in the video images included in the key frame set; A first sorting module, configured to sort the first numbers corresponding to the video images according to positions of the video images in the rainfall video, so as to obtain a first sorting result; A second sorting module, configured to sort the raindrop densities corresponding to the video images according to the positions of the video images in the rainfall video, so as to obtain a second sorting result; A third sorting module, configured to sort the raindrop speeds corresponding to the key frame set according to the positions of the key frame set in the rainfall video, so as to obtain a third sorting result; a fourth sorting module, configured to sort actual raindrop sizes of the raindrop areas contained in the video image according to the positions of the video images in the rainfall video, so as to obtain a fourth sorting result; A second acquisition module is used to input the first sorting result, the second sorting result, the third sorting result and the fourth sorting result into a pre-built rainfall prediction model; and output the predicted rainfall through the rainfall prediction model; Among them, the rainfall prediction model takes the first sorting result corresponding to the sample rainfall video, the second sorting result corresponding to the sample rainfall video, the third sorting result corresponding to the rainfall video and the fourth sorting result corresponding to the sample rainfall video as input, and takes the actual rainfall corresponding to the sample rainfall video as the training target for training.

Images