Hail observation method and device based on deep learning image recognition
Through deep learning image recognition technology, the problem that existing hail observation methods cannot measure the particle size and mass of hail particles is solved, and a comprehensive observation of hail particles is achieved, which improves the accuracy of hail prediction.
Patent Information
- Application Number
- CN202410995833.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-16
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
The existing hail observation methods cannot fully observe the strength and properties of hail particles. The sensor's observation area is small and it is impossible to measure the particle size and mass of hail.
Using deep learning image recognition method, we can obtain the image data of the plane by collecting hail captured by the camera device, identify hail particles, establish the relationship between the image coordinate system and the world coordinate system, calculate the diameter and mass density of hail, and output the hail quantity and average hail Diameter and mass density.
A more comprehensive observation of hail particles is achieved, and the intensity and properties of hail are more accurately measured, improving the accuracy of hail prediction.
Smart Images

Figure CN118941530A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of meteorological observation technology, and in particular to a hail observation method and system based on deep learning image recognition, a readable storage medium, and a computer device. Background Technology
[0002] Hail is a weather phenomenon that can cause significant damage to agriculture, production, and daily life. The observation of hail particles is of great significance for improving the accuracy of hail forecasting and assessing the impact of hail risks.
[0003] Current methods for observing hail primarily rely on sensors to monitor the occurrence, quantity, and intensity of hail. However, these sensors have very small observation areas, often only a circular sampling area with a diameter of a few centimeters. Furthermore, they typically cannot observe the size or mass of hailstones. Summary of the Invention
[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.
[0005] Therefore, the purpose of this invention is to provide a hail observation method and device based on deep learning image recognition, which can more comprehensively observe the intensity and properties of hail particles.
[0006] To achieve the above objectives, the first aspect of the present invention provides a hail observation method based on deep learning image recognition, comprising:
[0007] Acquire image data of the hail collection plane captured by the camera device;
[0008] Identify and label image data containing hail to obtain hail image data;
[0009] Identify hail particles in hail image data to obtain the number of hailstones;
[0010] Establish the relationship between the image coordinate system corresponding to the hail collection plane and the pixel coordinates in the world coordinate system of the corresponding camera device;
[0011] Based on the distance between two pixels in the horizontal direction of the hailstone diameter in the world coordinate system, the difference in the pixel coordinate relationship is calculated to obtain the difference representation of the pixel coordinate relationship;
[0012] Calculate the linear relationship between the actual spatial distance between two pixels on the hail collection plane and the spatial coordinates in the world coordinate system based on the installation information of the hail collection plane and the camera device.
[0013] Based on the linear relationship and combined with the pixel coordinates of hail particles in the image coordinate system, the hail diameter in the world coordinate system corresponding to the hail particles and the average hail diameter at the current moment are calculated.
[0014] The average mass density of hail is calculated based on the mass of hail particles collected by the hail collection plane.
[0015] The calculated number of hailstones, average hailstone diameter, and average hailstone mass density are output as the final results.
[0016] In the above technical solution, preferably, establishing the pixel coordinate relationship between the image coordinate system corresponding to the hail collection plane and the world coordinate system of the corresponding camera device includes:
[0017] With the center point of the image as the origin, and the horizontal direction to the right as the X-axis... i The direction, with the vertical upward as the Y-axis. i Establish the image coordinate system x in the direction of the image coordinate system. i -O i -y i ;
[0018] With the optical center O as the origin, and along the optical axis O i -O direction is z w The direction is defined by taking the x-axis as parallel to the X-axis in the image coordinate system. w The direction and y are parallel to the Y-axis in the image coordinate system. w Establish the world coordinate system x in the direction w -Oy w -z w ;
[0019] Based on the principle of triangle similarity, establish a point (x) in the world coordinate system. w ,y w , z w ) and pixel coordinates in the image coordinate system (x i ,y i The pixel coordinate relationship between the two points is expressed as follows:
[0020]
[0021] Among them, f x f y c x and c y These are the four internal parameters of the camera device, f x and f y Representing the image plane x i and y i The number of pixels corresponding to the focal length f along the coordinate axis; c x and c yThese refer to the center point of the photosensitive element and the origin O of the image plane coordinate system, respectively. i Between x i and y i The number of pixels offset along the coordinate axes; R is a 3×3 matrix representing the rotation relationship between the world coordinate system and the image coordinate system, vector t represents the translation relationship between the two coordinate systems, O T The meaning of filling the matrix;
[0022] Since there is no rotation relationship between the world coordinate system and the image coordinate system, the rotation matrix R in the formula can be taken as the identity matrix, and the expression is:
[0023] There is also a translation of -f along the Z-axis between the world coordinate system and the image coordinate system. There is no translation along the other two axes, X and Y. Therefore:
[0024] Equation (1) can be simplified from equations (3) and (4), that is:
[0025]
[0026] Where K is the intrinsic parameter matrix of the camera, expressed as:
[0027]
[0028] In any of the above technical solutions, preferably, image data containing hail is labeled using a B-Box selection method to obtain hail image data; and / or
[0029] The hail particles in the hail image data are identified, resulting in a hail set S consisting of N hail particles, specifically expressed as:
[0030] S = {S1, S2, S3, ..., S} N-1 , S N ]|S j =(x j,0 , x j,1 ,y i,0 ,y j,1 )} (1);
[0031] Among them, (x j,0 , x j,1 (y) represents the minimum and maximum y-coordinate values of the pixels occupied by the j-th hailstone, respectively; j,0 ,y j,1 ) represents the minimum and maximum y-coordinate values of the pixels occupied by the j-th hailstone; S j The coordinates are the minimum bounding box of the j-th hailstone identified by the deep learning algorithm.
[0032] In any of the above technical solutions, preferably, based on the distance between two pixels whose hail diameter is displayed horizontally in the world coordinate system, a horizontal difference operation is performed on the pixel coordinate relationship to obtain a difference representation of the pixel difference relationship, including:
[0033] Based on the difference between two pixels in the horizontal direction of the hailstone diameter displayed in the world coordinate system, a horizontal difference operation is performed on the pixel coordinate relationship to obtain the initial difference representation of the pixel coordinate relationship, expressed as:
[0034]
[0035] Based on the fact that the coordinates of two pixels in the hail collection plane and the world coordinate system have the same spatial width and spatial depth coordinates, the initial difference representation of the pixel coordinate relationship is simplified, resulting in the horizontal difference representation of the pixel coordinate relationship, expressed as:
[0036]
[0037] Expanding the horizontal difference representation of the pixel coordinate relationship, we obtain the difference representation of the pixel coordinate relationship, expressed as:
[0038]
[0039] In any of the above technical solutions, preferably, the linear relationship between the actual spatial distance between two pixels on the hail collection plane and the spatial coordinates of the world coordinate system is calculated based on the installation information of the hail collection plane and the camera device, including:
[0040] Based on the installation information of the hail collection plane and the camera device, the horizontal linear relationship between the actual spatial distance between two pixels on the horizontal direction of the hail collection plane and their spatial depth coordinates in the world coordinate system is calculated as follows:
[0041]
[0042] To simplify equation (10), let α = tanθ, β = -((H c- H t If )) / cosθ, then expression (10) simplifies to:
[0043] y w =αz w +β (11);
[0044] The vertical linearity relationship of the spatial depth coordinates in the world coordinate system is calculated based on the actual spatial distance in the vertical direction of the hail collection plane. Specifically:
[0045]
[0046] According to formulas (11) and (12), we get:
[0047] Substituting formula (13) into formula (9) yields the expression for the vertical linear relationship:
[0048]
[0049] In any of the above technical solutions, preferably, the hail diameter in the world coordinate system and the average hail diameter at the current moment are calculated based on a linear relationship and the pixel coordinates of the hail particles in the image coordinate system, including:
[0050] Based on horizontal and numerical linear relationships, the inverse proportional relationship between the spatial distance in the world coordinate system represented by a unit pixel between two horizontal pixels on the hail collection plane and the image width coordinate in the image coordinate system is calculated. Specifically:
[0051] set up And k2=-αf y -c y Substituting into formula (14), we obtain the inverse proportional relationship:
[0052]
[0053] Calculate the solution for the inverse proportional relationship based on the coordinates and dimensions of the four corner points of the hail collection plane in the image coordinate system:
[0054]
[0055] Where P i,1 (x P,i,1 ,y P,i,1 ), P i,2 (x P,i,2 ,y P,i,2 ), P i,3 (x P,i,3 ,y P,i,3 ) and P i,4 (x P,i,4 ,y P,i,4 P represents the coordinates of the four corner points of the hail collection plane. i,1 -P i,2 With P i,3 -P i,4 Parallelism, meaning the following relationship exists: y P,i,2 =y P,i,3 ,y P,i,1 =y P,i,4 ;
[0056] By combining the solution of the inverse proportional relationship with the pixel coordinates of the hail samples in the image coordinate system, the size of the hail particles in the world coordinate system is obtained, and the hail diameter is calculated accordingly.
[0057]
[0058] The average diameter of the hailstones at the current moment is calculated based on their diameter using the following expression:
[0059]
[0060] In any of the above technical solutions, preferably, the average mass density of hail is calculated based on the mass of hail particles collected by the hail collection plane, including:
[0061] The increase in hail mass at the current moment is calculated based on the cumulative mass of hail collected by the hail collection plane at the current moment, expressed as:
[0062]
[0063] M t Let this be the cumulative mass of hail collected by the hail collection plane at the current moment. This represents the current increment in hail mass.
[0064] The hail volume is calculated based on the hail mass increment and hail diameter, and the expression is:
[0065]
[0066] The average mass density of hail is calculated based on the hail volume and the increase in hail mass, and the expression is:
[0067] The second aspect of the present invention provides a hail observation system based on deep learning image recognition, comprising:
[0068] The acquisition module is configured to acquire image data of the hail collection plane captured by the camera device;
[0069] The hail image recognition module is configured to identify and label image data containing hail, thereby obtaining hail image data;
[0070] The hail count identification module is configured to identify hail particles in hail image data and obtain the hail count;
[0071] The pixel coordinate relationship establishment module is set up to establish the pixel coordinate relationship between the image coordinate system of the corresponding hail collection plane and the world coordinate system of the corresponding camera device.
[0072] The pixel coordinate representation module is set up to calculate the horizontal difference between two pixels based on the distance between them in the world coordinate system, which is displayed as the horizontal distance between the two pixels based on the hail diameter, and obtain the difference representation of the pixel coordinate relationship.
[0073] The coordinate linear relationship representation module is set up to calculate the linear relationship between the actual spatial distance between two pixels on the hail collection plane and the spatial coordinates of the world coordinate system, based on the hail collection plane and the installation information of the camera device.
[0074] The hail size calculation module is set up to calculate the hail diameter in the world coordinate system and the average hail diameter at the current moment based on a linear relationship and the pixel coordinates of the hail particles in the image coordinate system.
[0075] The hail density calculation module is set up to calculate the average mass density of hail based on the mass of hail particles collected by the hail collection plane.
[0076] The results output module is set to output the calculated number of hailstones, average hailstone diameter, and average hailstone mass density as the final results.
[0077] The third aspect of the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the hail observation method based on deep learning image recognition provided in the first aspect of the present invention.
[0078] The fourth aspect of the present invention provides a computer device, including a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the steps of the hail observation method based on deep learning image recognition provided in the first aspect of the present invention.
[0079] Compared with existing technologies, the hail observation method and equipment based on deep learning image recognition provided by this invention have the advantage of being able to observe the intensity and properties of hail particles more comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:
[0081] Figure 1 A flowchart of the hail observation method according to an embodiment of the present invention is shown;
[0082] Figure 2 A flowchart of step S4 as described in an embodiment of the present invention is shown;
[0083] Figure 3A flowchart of step S5 as described in an embodiment of the present invention is shown;
[0084] Figure 4 A flowchart of step S6 as described in an embodiment of the present invention is shown;
[0085] Figure 5 A flowchart of step S7 as described in an embodiment of the present invention is shown;
[0086] Figure 6 A flowchart of step S8 as described in an embodiment of the present invention is shown;
[0087] Figure 7 A structural block diagram of the hail observation system according to an embodiment of the present invention is shown;
[0088] Figure 8 A schematic diagram showing the positional relationship between the camera device and the hail collection plane according to an embodiment of the present invention is shown;
[0089] Figure 9 This diagram illustrates the coordinate points of the hail collection plane in the image coordinate system according to an embodiment of the present invention.
[0090] Figure 10 A schematic diagram of the image coordinate system and world coordinate system involved in the embodiments of the present invention is shown. DETAILED DESCRIPTION
[0091] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0092] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0093] like Figures 1 to 10 As shown, a hail observation method based on deep learning image recognition according to an embodiment of the present invention includes:
[0094] S1, acquire image data of the hail collection plane captured by the camera device;
[0095] In this step, such as Figure 2 As shown, the high-definition camera is mounted on a pole, and the optical center of its camera lens is H meters above the plumb bob on the ground. c The camera's downward angle is θ (horizontal is 0°). The hail collection platform is parallel to the ground and at a height H above the ground. tThe hail collection platform is a square with dimensions W×W. The camera takes one image of the hail collection surface every minute at second 0.
[0096] S2, identify and label image data containing hail to obtain hail image data;
[0097] S3, identify hail particles in hail image data and obtain the number of hailstones;
[0098] In steps S2 and S3, a Transformer-based ConvNext network is used to detect the target. This model is trained using a dataset of hundreds of high-quality hail image annotations, with each hailstone labeled using a bounding box (BBox). The training dataset contains nearly ten thousand hailstone samples. Based on the captured images, a hailstone set S consisting of N hailstones can be calculated and identified.
[0099] S4, Establish the relationship between the image coordinate system corresponding to the hail collection plane and the pixel coordinates in the world coordinate system of the corresponding camera device;
[0100] S5, based on the distance between two pixels in the horizontal direction of the hail diameter displayed in the world coordinate system, calculate the difference in the pixel coordinate relationship in the horizontal direction to obtain the difference representation of the pixel coordinate relationship;
[0101] S6, calculate the linear relationship between the actual spatial distance between two pixels on the hail collection plane and the spatial coordinates of the world coordinate system based on the hail collection plane and the installation information of the camera device;
[0102] S7. Based on the linear relationship and combined with the pixel coordinates of the hail particles in the image coordinate system, calculate the hail diameter of the hail particles in the world coordinate system and the average hail diameter at the current moment.
[0103] S8, calculate the average mass density of hail based on the mass of hail particles collected by the hail collection plane;
[0104] S9 outputs the calculated number of hailstones, average hailstone diameter, and average hailstone mass density as the final results.
[0105] In step S9, the data obtained from the above steps, such as the number of hailstones, hailstone size, and hailstone mass density, are processed to obtain the following results:
[0106]
[0107] If N = 0, then the last two variables are filled with hyphens, and the final result is:
[0108] O t =[0,-,-] (24).
[0109] like Figure 2 and Figure 10 As shown, in the above embodiment, preferably, S4, establishing the pixel coordinate relationship between the image coordinate system corresponding to the hail collection plane and the world coordinate system of the corresponding camera device includes:
[0110] S41, with the center point of the image as the origin and the horizontal direction to the right as the X-axis x i The direction, with the vertical upward as the Y-axis. i Establish the image coordinate system x in the direction of the image coordinate system. i -O i -y i ;
[0111] S42, with the optical center O as the origin, and along the optical axis O i -O direction is z w The direction is defined by taking the x-axis as parallel to the X-axis in the image coordinate system. w The direction and y are parallel to the Y-axis in the image coordinate system. w Establish the world coordinate system x in the direction w -Oy w -z w ;
[0112] S43, Establishing a point (x) in the world coordinate system based on the principle of triangle similarity. w ,y w , z w ) and pixel coordinates in the image coordinate system (x i ,y i The pixel coordinate relationship between the two points is expressed as follows:
[0113]
[0114] Among them, f x f y c x and c y These are the four internal parameters of the camera device, f x and f y Representing the image plane x i and y i The number of pixels corresponding to the focal length f along the coordinate axis; c x and c y These refer to the center point of the photosensitive element and the origin O of the image plane coordinate system, respectively. i Between x i and y i The number of pixels offset along the coordinate axes; R is a 3×3 matrix representing the rotation relationship between the world coordinate system and the image coordinate system, vector t represents the translation relationship between the two coordinate systems, O TThe meaning of filling the matrix;
[0115] Since there is no rotation relationship between the world coordinate system and the image coordinate system, the rotation matrix R in the formula can be taken as the identity matrix, and the expression is:
[0116] There is also a translation of -f along the Z-axis between the world coordinate system and the image coordinate system. There is no translation along the other two axes, X and Y. Therefore:
[0117] Equation (1) can be simplified from equations (3) and (4), that is:
[0118]
[0119] Where K is the intrinsic parameter matrix of the camera, expressed as:
[0120]
[0121] In any of the above embodiments, preferably, image data containing hail is labeled using a B-Box selection method to obtain hail image data; and / or
[0122] The hail particles in the hail image data are identified, resulting in a hail set S consisting of N hail particles, specifically expressed as:
[0123] s = {[S1, S2, S3, ..., S...} N-1 ,S N ]|S j =(x j,0 , x j,1 ,y j,0 ,y j,1 )} (1);
[0124] Among them, (x j,0 , x j,1 (y) represents the minimum and maximum y-coordinate values of the pixels occupied by the j-th hailstone, respectively; j,0 ,y j,1 ) represents the minimum and maximum y-coordinate values of the pixels occupied by the j-th hailstone; S j The coordinates are the minimum bounding box of the j-th hailstone identified by the deep learning algorithm.
[0125] like Figure 3 As shown, in any of the above embodiments, preferably, S5, based on the distance between two pixels whose hail diameter is displayed horizontally in the world coordinate system, a horizontal difference operation is performed on the pixel coordinate relationship to obtain a difference representation of the pixel difference relationship, including:
[0126] S51, based on the difference between two pixels whose hailstone diameter is displayed horizontally in the world coordinate system, the pixel coordinate relationship is adjusted horizontally (x-axis). w The difference operation yields the initial difference representation of the pixel coordinate relationship, expressed as:
[0127]
[0128] S52, based on the fact that the coordinates of two pixels in the hail collection plane and the world coordinate system have the same spatial width and spatial depth coordinates, the initial difference representation of the pixel coordinate relationship is simplified, resulting in the horizontal difference representation of the pixel coordinate relationship, expressed as:
[0129]
[0130] In this step, only the distance between two horizontal points on the hail collection plane in the image is considered; that is, the line connecting these two points is parallel to the upper and lower edges of the square collection plane both in the image and on the actual collection plane. This is combined with the installation structure of the hail observation device and... Figure 9 The characteristics of the world coordinate system shown indicate that two points have the same y-coordinate. w Coordinates and z w coordinate.
[0131] S53, expand the horizontal difference representation of the pixel coordinate relationship to obtain the difference representation of the pixel coordinate relationship, the expression is:
[0132]
[0133] like Figure 4 As shown, in any of the above embodiments, preferably, S6, calculating the linear relationship between the actual spatial distance between two pixels on the hail collection plane and the spatial coordinates of the world coordinate system based on the installation information of the hail collection plane and the camera device includes:
[0134] S61, based on the hail collection plane and the installation information of the camera device, calculate the horizontal linear relationship between the actual spatial distance between two pixels in the horizontal direction of the hail collection plane and their spatial depth coordinates in the world coordinate system, specifically:
[0135]
[0136] That is, the actual spatial distance represented by a unit pixel between two horizontal points on the hail collection plane and the spatial depth coordinates z of these two points. w The relationship is linearly related. Considering the structural characteristics of the hail collection plane device and the installation angle of the camera, the coordinates of the points on the hail collection platform have the relationship shown in the above formula (10).
[0137] To simplify equation (10), let α = tanθ, β = -((H c- H t If )) / cosθ, then expression (10) simplifies to:
[0138] y w =αz w +β(11);
[0139] S62, the vertical linearity relationship of the spatial depth coordinates in the world coordinate system is calculated based on the actual spatial distance in the vertical direction of the hail collection plane. Specifically:
[0140]
[0141] In this step, the Y and Z coordinates of points on the hail collection plane in the world coordinate system have a linear relationship. Considering the difference between two points on the vertical direction of the image corresponding to the hail collection plane, these two points have the same X value, and the y-direction relationship can be extracted using formulas 5 and 6.
[0142] According to formulas (11) and (12), we get:
[0143] Substituting formula (13) into formula (9) yields the expression for the vertical linear relationship:
[0144]
[0145] like Figure 5 As shown, in any of the above embodiments, preferably, S7, based on the linear relationship and combined with the pixel coordinates of the hail particles in the image coordinate system, the hail diameter corresponding to the hail particles in the world coordinate system and the average hail diameter at the current moment are calculated, including:
[0146] S71, based on horizontal and numerical linear relationships, calculate the inverse proportional relationship between the spatial distance in the world coordinate system represented by a unit pixel between two horizontal pixels on the hail collection plane and the image width coordinate in the image coordinate system, specifically:
[0147] set up And k2=-αf y -c y Substituting into formula (14), we obtain the inverse proportional relationship:
[0148]
[0149] S72, calculate the solution for the inverse proportional relationship based on the coordinates and size W of the four corner points of the hail collection plane in the image coordinate system:
[0150]
[0151] like Figure 9 As shown, where P i,1 (x P,i,1 ,y P,i,1 ), P i,2 (x P,i,2 ,y P,i,2 ), P i,3 (x P,i,3 ,y P,i,3 ) and P i,4 (x P,i,4 ,y P,i,4 P represents the coordinates of the four corner points of the hail collection plane. i,1 -P i,2 With P i,3 -P i,4 Parallelism, meaning the following relationship exists: y P,i,2 =y P,i,3 ,y P,i,1 =y P,i,4 ;
[0152] S73, based on the solution of the inverse proportional relationship and the pixel coordinates of the hail sample in the image coordinate system, the size of the hail particles in the world coordinate system is obtained, thereby calculating the hail diameter, specifically:
[0153]
[0154] S74, calculate the average hail diameter at the current moment based on the hail's diameter, using the following expression:
[0155]
[0156] like Figure 6 As shown, in any of the above embodiments, preferably, S8, calculating the average mass density of hail based on the mass of hail particles collected by the hail collection plane includes:
[0157] S81, calculate the hail mass increment at the current moment based on the cumulative hail mass collected by the hail collection plane at the current moment, expressed as:
[0158]
[0159] M t Let this be the cumulative mass of hail collected by the hail collection plane at the current moment. This represents the current increment in hail mass.
[0160] S82, the hail volume is calculated based on the hail mass increment and hail diameter, and the expression is:
[0161]
[0162] S83, the average mass density of hail is calculated based on the hail volume and the increase in hail mass, and the expression is:
[0163] like Figure 7 As shown, a hail observation system 100 based on deep learning image recognition according to another embodiment of the present invention includes:
[0164] The acquisition module 10 is configured to acquire image data of the hail collection plane captured by the camera device;
[0165] Hail image recognition module 20 is configured to identify and label image data containing hail to obtain hail image data;
[0166] The hail quantity recognition module 30 is configured to identify hail particles in hail image data and obtain the hail quantity;
[0167] The pixel coordinate relationship establishment module 40 is set to establish the pixel coordinate relationship between the image coordinate system of the corresponding hail collection plane and the world coordinate system of the corresponding camera device.
[0168] The pixel coordinate representation module 50 is configured to calculate the horizontal difference between two pixels based on the distance between them in the world coordinate system, which is displayed as the horizontal direction based on the hail diameter, and obtain a difference representation of the pixel coordinate relationship.
[0169] The coordinate linear relationship representation module 60 is configured to calculate the linear relationship between the actual spatial distance between two pixels on the hail collection plane and the spatial coordinates of the world coordinate system based on the hail collection plane and the installation information of the camera device.
[0170] The hail size calculation module 70 is configured to calculate the hail diameter in the world coordinate system and the average hail diameter at the current moment based on a linear relationship and the pixel coordinates of the hail particles in the image coordinate system.
[0171] The hail density calculation module 80 is set to calculate the average mass density of hail based on the mass of hail particles collected by the hail collection plane.
[0172] The result output module 90 is set to output the calculated number of hailstones, the average diameter of hailstones, and the average mass density of hailstones as the final results.
[0173] Based on the above, Figure 5 and Figure 6Accordingly, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the hail observation method based on deep learning image recognition of any of the above embodiments.
[0174] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.
[0175] Based on the above, Figure 5 and Figure 6 The method shown, and Figure 7 To achieve the above objectives, the present application also provides a computer device, including a storage medium and a processor, as shown in the virtual device embodiment. The storage medium is used to store a computer program; the processor is used to execute the computer program to implement the steps of the hail observation method based on deep learning image recognition of any of the above embodiments.
[0176] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.
[0177] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.
[0178] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software within the physical device.
[0179] In this invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more unless otherwise explicitly defined. The terms "install," "connect," "link," and "fix" should be interpreted broadly. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; "link" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0180] In the description of this invention, it should be understood that the terms "upper," "lower," "left," "right," "front," "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or unit referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0181] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0182] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.
Claims
1. A hail observation method based on deep learning image recognition, characterized in that: include: Acquire image data of the hail collection plane captured by a camera device; Identifying and marking the image data containing hail to obtain hail image data; Identifying hail particles in the hail image data to obtain the number of hail; Establishing a coordinate relationship between an image coordinate system corresponding to the hail collection plane and a pixel point in a world coordinate system corresponding to the camera device; Based on the fact that the diameter of the hailstone is displayed as the distance between two pixels in the horizontal direction in the world coordinate system, performing a horizontal difference calculation on the coordinate relationship of the pixel points to obtain a difference representation of the coordinate relationship of the pixel points; Calculating the linear relationship between the actual spatial distance between the two pixel points on the hail collection plane and the spatial coordinates of the world coordinate system according to the installation information of the hail collection plane and the camera device; According to the linear relationship and in combination with the pixel coordinates of the hail particles in the image coordinate system, the hail diameter corresponding to the hail particles in the world coordinate system and the average hail diameter at the current moment are calculated; The average mass density of the hail is calculated according to the mass of the hail particles collected by the hail collecting plane; The calculated hail quantity, hail average diameter and hail average mass density are output as final results.
2. The hail observation method based on deep learning image recognition according to claim 1, characterized in that: Establishing a coordinate relationship between an image coordinate system corresponding to the hail collection plane and a pixel point in a world coordinate system corresponding to the camera device includes: The center of the image is taken as the origin, and the horizontal right direction is taken as the X-axis x i Direction, with vertical upward as the Y axis y i direction, establish the image coordinate system x i -O i -y i ; With the optical center O as the origin and along the optical axis O i -O direction is z w The direction is parallel to the X axis in the image coordinate system as x w The direction is parallel to the Y axis in the image coordinate system as y w The world coordinate system is established in the direction x w -Oy w -z w ; Based on the triangle similarity principle, the point (x w ,y w , z w ) and the pixel coordinates (x i ,y i ) between the pixel coordinates, the specific expression is: Among them, f x 、f y 、c x and c y are the four internal parameters of the camera device, f x and f y Denote the image plane x i and i The number of pixels corresponding to the focal length f in the coordinate axis direction; c x and c y They refer to the center point of the photosensitive element and the origin of the image plane coordinates O. i Between x i and i The number of pixels offset in the coordinate axis direction; R is a 3×3 matrix, representing the rotation relationship between the world coordinate system and the image coordinate system, and the vector t represents the translation relationship between the two coordinate systems. T To fill in the meaning of the matrix; Since there is no rotation relationship between the world coordinate system and the image coordinate system, the rotation matrix R in the formula can be obtained as the unit matrix, and the expression is: There is also a translation of -f in the Z-axis direction between the world coordinate system and the image coordinate system, and there is no translation in the other two X-axis and Y-axis directions. It can be obtained: Formula (1) can be simplified by formula (3) and formula (4), namely: Where K is the intrinsic parameter matrix of the camera, expressed as:
3. The hail observation method based on deep learning image recognition according to claim 2, characterized in that: Marking the image data containing hail using a B-Box selection method to obtain hail image data; and / or The hail particles in the hail image data are identified to obtain a hail set S consisting of a total of N hail particles. The specific expression is: S={[S1,S2,S3,...,S N-1 ,S N ]|S j =(x j,0 ,x j,1 ,y j,0 ,y j,1 )} (1); Among them, (x j,0 , x j,1 ) represent the minimum and maximum y-coordinate values of the pixels occupied by the jth hailstone particle; (y j,0 ,y j,1 ) respectively represent the minimum and maximum y coordinate values of the pixels occupied by the jth hailstone particle; S j The coordinates of the minimum bounding box of the j-th hailstone identified by the deep learning algorithm.
4. The hail observation method based on deep learning image recognition according to claim 3 is characterized in that: Based on the fact that the hailstone diameter is displayed as the distance between two pixels in the horizontal direction in the world coordinate system, a horizontal difference operation is performed on the pixel coordinate relationship to obtain a difference representation of the pixel difference relationship, including: Based on the fact that the hailstone diameter is displayed as the difference between two horizontal pixels in the world coordinate system, a horizontal difference operation is performed on the pixel coordinate relationship to obtain an initial difference representation of the pixel coordinate relationship, which is expressed as follows: Based on the fact that the coordinates of the two pixels in the hail collection plane and the world coordinate system have the same spatial width coordinate and spatial depth coordinate, the initial difference representation of the pixel coordinate relationship is simplified to obtain the horizontal difference representation of the pixel coordinate relationship, which is expressed as follows: The horizontal difference representation of the pixel point coordinate relationship is expanded and decomposed to obtain the difference representation of the pixel point coordinate relationship, and the expression is:
5. The hail observation method based on deep learning image recognition according to claim 4 is characterized in that: Calculating the linear relationship between the actual spatial distance between the two pixel points on the hail collection plane and the spatial coordinates of the world coordinate system according to the installation information of the hail collection plane and the camera device includes: The horizontal linear relationship between the actual spatial distance between the two pixel points in the horizontal direction of the hail collection plane and the spatial depth coordinates in the world coordinate system is calculated according to the installation information of the hail collection plane and the camera device, specifically: To simplify equation (10), let α = tanθ, β = -((H c- H t )) / cosθ, then expression (10) is simplified to: y w =αz w +β (11); The vertical linear relationship of the spatial depth coordinates in the world coordinate system is calculated according to the actual spatial distance in the vertical direction of the hail collection plane, specifically: According to formula (11) and formula (12), we can get: Substituting formula (13) into formula (9) yields the expression of the vertical linear relationship:
6. The hail observation method based on deep learning image recognition according to claim 5, characterized in that: According to the linear relationship and in combination with the pixel coordinates of the hail particles in the image coordinate system, the hail diameter corresponding to the hail particles in the world coordinate system and the average hail diameter at the current moment are calculated, including: The inverse proportional relationship between the spatial distance in the world coordinate system represented by a unit pixel between two horizontal pixel points on the hail collection plane and the image width coordinate in the image coordinate system is calculated according to the horizontal linear relationship and the numerical linear relationship, specifically: set up and k2 = -αf y -c y Substituting into formula (14) we obtain the inverse proportional relationship: The solution of the inverse proportional relationship is calculated based on the coordinates and sizes of the four corner points of the hail collection plane in the image coordinate system: Where P i,1 (x P,i,1 ,y P,i,1 ), P i,2 (x P,i,2 ,y P,i,2 ), P i,3 (x P,i,3 ,y P,i,3 ) and P i,4 (x P,i,4 ,y P,i,4 ) represent the coordinates of the four corner points of the hail collection plane, P i,1 -P i,2 With P i,3 -P i,4 Parallel, that is, the following relationship exists: P,i,2 =y P,i,3 ,y P,i,1 =y P,i,4 ; According to the solution of the inverse proportional relationship and the pixel coordinates of the hail sample in the image coordinate system, the size of the hail particle in the world coordinate system is obtained, so as to calculate the hail diameter, specifically: The average diameter of hail at the current moment is calculated based on the diameter of hail. The expression is:
7. The hail observation method based on deep learning image recognition according to claim 6, characterized in that: The average mass density of hail is calculated according to the mass of hail particles collected by the hail collecting plane, including: The hail mass increment at the current moment is calculated according to the cumulative mass of hail collected by the hail collection plane at the current moment, and the expression is: M t is the cumulative mass of hail collected by the hail collecting plane at the current moment, is the mass increment of the hail at the current moment; The hail volume is calculated according to the hail mass increment and the hail diameter, and the expression is: The average mass density of the hail is calculated according to the hail volume and the hail mass increment, and the expression is:
8. A hail observation system based on deep learning image recognition, characterized in that: include: An acquisition module is configured to acquire image data of the hail collection plane captured by a camera device; A hail image recognition module is configured to recognize and mark the image data containing hail to obtain hail image data; A hail quantity recognition module is configured to recognize hail particles in the hail image data to obtain the hail quantity; A pixel point coordinate relationship establishing module, configured to establish a pixel point coordinate relationship in an image coordinate system corresponding to the hail collection plane and a world coordinate system corresponding to the camera device; A pixel point coordinate representation module is configured to calculate the horizontal difference of the pixel point coordinate relationship based on the hail diameter being displayed as the distance between two pixels in the horizontal direction in the world coordinate system, so as to obtain the difference representation of the pixel point coordinate relationship; a coordinate linear relationship representation module, configured to calculate the linear relationship between the actual spatial distance between the two pixel points on the hail collection plane and the spatial coordinates of the world coordinate system according to the installation information of the hail collection plane and the camera device; A hail size calculation module is configured to calculate the hail diameter of the hail particles in the world coordinate system and the average hail diameter at the current moment according to the linear relationship and in combination with the pixel coordinates of the hail particles in the image coordinate system; A hail density calculation module is configured to calculate the average mass density of hail according to the mass of hail particles collected by the hail collection plane; The result output module is configured to output the calculated number of hailstones, the average diameter of hailstones and the average mass density of hailstones as final results.
9. A readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of the hail observation method based on deep learning image recognition as described in any one of claims 1 to 7 are implemented.
10. A computer device, characterized in that: It includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the steps of the hail observation method based on deep learning image recognition as described in any one of claims 1 to 7.
