Falling object prediction method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a method for predicting falling objects, the method comprising: obtaining historical image data and historical meteorological data of each area to be tested; calculating the historical falling object data of each area to be tested according to the historical image data and historical meteorological data corresponding to each area to be tested; generating a target adjacency matrix between each area to be tested according to the positional relationship between each area to be tested and the historical falling object data of each area to be tested; based on the target adjacency matrix, predicting the falling objects of each area to be tested, and obtaining the falling object prediction results of each area to be tested. The present invention uses the target adjacency matrix to predict the falling objects of each area to be tested, and the global meteorological influence between the areas can be considered, so that the falling object prediction results are more accurate, and then sanitation workers can be arranged in advance to the areas with more falling object predictions, thereby improving the cleaning efficiency of urban falling objects.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a falling object prediction method, device, electronic equipment and storage medium. Background Art

[0002] In urban management, urban sanitation work cannot be underestimated. There will be fallen objects such as paper scraps, fallen leaves, and plastic bags in the city. Fallen objects will drift with the flow of air. For example, fallen leaves or dust will be blown to other areas by the wind. When sanitation workers are cleaning, they often clean the areas with more fallen objects first to be most efficient, so as to prevent the fallen objects from being blown to other areas by the wind. The existing method can use monitoring images to dispatch sanitation workers to the areas with the most fallen objects in real time. However, there is a lag in the arrival of sanitation workers. When sanitation workers arrive, the fallen objects may have drifted with the flow of air. Therefore, the existing urban fallen objects cleaning has the problem of low efficiency. Summary of the invention

[0003] The embodiment of the present invention provides a method for predicting falling objects, aiming to solve the problem of low cleaning efficiency of existing falling objects in cities. The historical falling object data of each area to be tested is calculated by using the historical image data and historical meteorological data of each area to be tested, and then the target adjacency matrix is ​​generated to globally express each area to be tested. The falling objects of each area to be tested are predicted using the target adjacency matrix, and the global meteorological influence between areas can be considered, so that the falling object prediction result is more accurate, and sanitation workers can be arranged in advance to areas with more falling object predictions, thereby improving the cleaning efficiency of urban falling objects.

[0004] In a first aspect, an embodiment of the present invention provides a falling object prediction method, the method comprising:

[0005] Obtain historical image data and historical meteorological data for each area to be tested;

[0006] Calculate the historical falling object data of each of the areas to be tested according to the historical image data and the historical meteorological data corresponding to each of the areas to be tested;

[0007] Generate a target adjacency matrix between the areas to be tested according to the positional relationship between the areas to be tested and the historical falling object data of the areas to be tested;

[0008] Based on the target adjacency matrix, the falling objects in each of the areas to be tested are predicted to obtain the falling object prediction results for each of the areas to be tested.

[0009] Optionally, the calculating, according to the historical image data and the historical meteorological data corresponding to each of the areas to be tested, the historical falling object data of each area to be tested includes:

[0010] The historical image data and the historical meteorological data are sampled with the same time point as the starting point and the same time step to obtain N frames of sample image data and corresponding N frames of sample meteorological data, each of the sample image data corresponds to one frame of sample meteorological data;

[0011] Extracting spatial features of the falling objects in the sample image data of the nth frame to obtain spatial features of the falling objects in the sample image data of the nth frame, where n is a positive integer less than or equal to N;

[0012] Extracting aerodynamic features of the sample meteorological data of the nth frame to obtain aerodynamic features of the sample meteorological data of the nth frame;

[0013] The spatial features of the falling objects in the sample image data of the nth frame are combined with the aerodynamic features of the corresponding sample meteorological data of the nth frame to obtain the falling object feature data of the nth frame;

[0014] Arrange the N frames of the falling object feature data in the time dimension to obtain the historical falling object data of the area to be tested.

[0015] Optionally, extracting spatial features of the falling objects in the sample image data of the nth frame to obtain the spatial features of the falling objects in the sample image data of the nth frame includes:

[0016] Performing target detection on the falling objects in the sample image data of the nth frame, and obtaining the spatial distribution characteristics of the falling objects in the sample image data of the nth frame according to the target detection result;

[0017] Performing image segmentation on the falling objects in the sample image data of the nth frame, and obtaining the spatial proportion characteristics of the falling objects in the sample image data of the nth frame according to the image segmentation result;

[0018] The spatial distribution characteristics of the falling objects in the sample image data of the nth frame are spliced ​​with the spatial proportion characteristics of the falling objects in the sample image data of the nth frame to obtain the spatial characteristics of the falling objects in the sample image data of the nth frame.

[0019] Optionally, generating a target adjacency matrix according to the positional relationship between the areas to be tested and the historical falling object data of the areas to be tested includes:

[0020] Determining the adjacency relationship of each of the areas to be tested according to the positional relationship of each of the areas to be tested;

[0021] According to the adjacency relationship, an initial adjacency matrix between the regions to be tested is established;

[0022] The historical falling object data of each of the areas to be tested is added to the corresponding matrix unit in the initial adjacency matrix to obtain the target adjacency matrix between each of the areas to be tested.

[0023] Optionally, predicting the falling objects in each of the areas to be tested based on the target adjacency matrix to obtain the falling object prediction results for each of the areas to be tested includes:

[0024] Extracting global spatial features of the target adjacency matrix in the time dimension to obtain global spatial features corresponding to different time points;

[0025] Based on the time dependency relationship between the global spatial features corresponding to the different time points, the time features of the global spatial features corresponding to the different time points are encoded to obtain global spatiotemporal features;

[0026] Linear regression is performed on the global time feature to obtain the falling object prediction results of each of the areas to be tested.

[0027] Optionally, encoding the time features of the global spatial features corresponding to the different time points based on the time dependency between the global spatial features corresponding to the different time points to obtain the global spatiotemporal features includes:

[0028] Segmenting the time dimension of the target adjacency matrix to obtain K time periods, where N is an integer multiple of K;

[0029] Expand the global space features corresponding to the different time points in one dimension to obtain a global space vector;

[0030] According to the time dependency between the global space vectors in each time period, vector encoding is performed on each time period to obtain K spatiotemporal sub-features;

[0031] The K spatiotemporal sub-features are clustered by a preset clustering method to obtain a cluster center as a global spatiotemporal feature.

[0032] Optionally, clustering the K spatiotemporal sub-features by a preset clustering method to obtain a cluster center as a global spatiotemporal feature includes:

[0033] According to the preset path coefficient, the path result from the kth spatiotemporal sub-feature to the jth cluster space is calculated;

[0034] According to the path result from the kth spatiotemporal sub-feature to the jth clustering space, the clustering space parameters of the jth clustering space are calculated as the global spatiotemporal feature.

[0035] In a second aspect, an embodiment of the present invention provides a falling object prediction device, the device comprising:

[0036] An acquisition module is used to acquire historical image data and historical meteorological data of each area to be tested;

[0037] A calculation module, used for calculating the historical falling object data of each of the areas to be measured according to the historical image data and the historical meteorological data corresponding to each of the areas to be measured;

[0038] A generating module, used for generating a target adjacency matrix between each of the areas to be tested according to the positional relationship between each of the areas to be tested and the historical falling object data of each of the areas to be tested;

[0039] The prediction module is used to predict the falling objects in each of the areas to be tested based on the target adjacency matrix to obtain the falling object prediction results of each of the areas to be tested.

[0040] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the falling object prediction method provided in the embodiment of the present invention when executing the computer program.

[0041] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the falling object prediction method provided in the embodiment of the invention are implemented.

[0042] In an embodiment of the present invention, historical image data and historical meteorological data of each area to be tested are obtained; historical falling object data of each area to be tested is calculated based on the historical image data and historical meteorological data corresponding to each area to be tested; a target adjacency matrix between each area to be tested is generated based on the positional relationship between each area to be tested and the historical falling object data of each area to be tested; based on the target adjacency matrix, the falling objects of each area to be tested are predicted to obtain the falling object prediction results of each area to be tested. The historical falling object data of each area to be tested is calculated through the historical image data and historical meteorological data of each area to be tested, and then the target adjacency matrix is ​​generated to express each area to be tested globally. The target adjacency matrix is ​​used to predict the falling objects of each area to be tested, and the global meteorological influence between areas can be considered, so that the falling object prediction results are more accurate, and sanitation workers can be arranged in advance to areas with more falling object predictions, thereby improving the cleaning efficiency of urban falling objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a flow chart of a falling object prediction method provided by an embodiment of the present invention;

[0045] Figure 2 is a structural schematic diagram of a falling object prediction device provided by an embodiment of the present invention;

[0046] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] See also Figure 1 , Figure 1 is a flow chart of a falling object prediction method provided by an embodiment of the present invention. Figure 1 As shown, the falling object prediction method includes the following steps:

[0049] 101. Obtain historical image data and historical meteorological data of each area to be tested.

[0050] In the embodiment of the present invention, the above-mentioned areas to be tested are connected areas, that is, each area to be tested is directly adjacent to at least one other area to be tested. The above-mentioned areas to be tested can be divided according to the cleaning jurisdiction of the sanitation staff, and each area to be tested corresponds to a cleaning jurisdiction.

[0051] One or more image acquisition devices, such as cameras, are provided in each area to be tested. The image acquisition devices are used to collect image data of the corresponding area to be tested. The above historical image data may be image data of the corresponding area to be tested at the current time point or before a specified time point. The specified time point may be before the current time point. The above historical meteorological data may be meteorological data of the corresponding area to be tested at the current time point or before a specified time point. The meteorological data may be provided by the meteorological department of the area to be tested.

[0052] 102. Calculate the historical falling object data of each area to be measured based on the historical image data and historical meteorological data corresponding to each area to be measured.

[0053] In an embodiment of the present invention, the spatial data of falling objects corresponding to each historical moment can be calculated based on the historical image data, such as the spatial data such as the distribution position of falling objects, the coverage area of ​​falling objects, and the area ratio of falling objects. The meteorological impact data corresponding to each historical moment can be calculated based on the historical meteorological data, such as the meteorological impact data such as wind direction, wind force, and air humidity. Among them, the meteorological impact data can be used as an influencing factor of the spatial data of falling objects, that is, the distribution position of falling objects, the coverage area of ​​falling objects, and the area ratio of falling objects will be affected by meteorological data such as wind direction, wind force, and air humidity. The historical falling object data of each area to be measured can be obtained based on the spatial data of falling objects corresponding to each historical moment and the meteorological impact data corresponding to the same historical moment, and the above-mentioned historical falling object data includes the spatial data of falling objects corresponding to each historical moment and the meteorological impact data corresponding to each historical moment.

[0054] 103. Generate a target adjacency matrix between each area to be tested according to the positional relationship between each area to be tested and the historical falling object data of each area to be tested.

[0055] In an embodiment of the present invention, each area to be tested is adjacent to at least one other area to be tested, which can be understood as the two areas to be tested being adjacent to each other or the two areas to be tested being connected through another area to be tested. The positional relationship between the above-mentioned areas to be tested includes an adjacent relationship and an orientation relationship. Each area to be tested can be marked on a map, with each area to be tested as a matrix unit, two adjacent areas to be tested as two adjacent matrix units, two areas to be tested adjacent to the left and the right as two adjacent matrix units to the left and the right, and two areas to be tested adjacent to one above and one below as two adjacent matrix units to the above and one below, and blank matrix units are filled in the positions where there are no corresponding areas to be tested to obtain an initial matrix. The historical falling object data of each area to be tested is added to the corresponding matrix unit to obtain a target adjacency matrix between each area to be tested. The target adjacency matrix includes a spatial dimension and a time dimension. The spatial dimension is determined according to the number of areas to be tested, and the time dimension is determined according to the historical moments contained in the historical falling object data.

[0056] The above target adjacency matrix contains the global spatial relationship of each area to be tested, and the matrix unit contains the local spatial relationship of the corresponding area to be tested. The global spatial relationship can be used to describe the global distribution of falling objects in all areas to be tested, and the local spatial relationship can be used to describe the local distribution of falling objects in a certain area to be tested. At the same time, the target adjacency matrix also contains a time attribute, which describes the implicit relationship of the global spatial relationship of each area to be tested changing with time and the implicit relationship of the local spatial relationship of the area to be tested changing with time.

[0057] 104. Based on the target adjacency matrix, the falling objects in each area to be tested are predicted to obtain the falling object prediction results of each area to be tested.

[0058] In the embodiment of the present invention, the target adjacency matrix includes the global spatial relationship and time attribute of each area to be tested, and the matrix unit includes the local spatial relationship of the corresponding area to be tested. The implicit relationship of the global spatial relationship of each area to be tested changing with time and the implicit relationship of the local spatial relationship of the area to be tested changing with time can be used to predict the distribution of falling objects in each area to be tested in the future, so as to obtain the falling object prediction results of each area to be tested.

[0059] The predicted results of falling objects in the above-mentioned areas to be tested can be used to guide the work of sanitation workers, and sanitation workers can be arranged in advance to clean up areas with more falling objects predicted, so as to prevent falling objects from drifting to other areas due to meteorological reasons.

[0060] In an embodiment of the present invention, historical image data and historical meteorological data of each area to be tested are obtained; historical falling object data of each area to be tested is calculated based on the historical image data and historical meteorological data corresponding to each area to be tested; a target adjacency matrix between each area to be tested is generated based on the positional relationship between each area to be tested and the historical falling object data of each area to be tested; based on the target adjacency matrix, the falling objects of each area to be tested are predicted to obtain the falling object prediction results of each area to be tested. The historical falling object data of each area to be tested is calculated through the historical image data and historical meteorological data of each area to be tested, and then the target adjacency matrix is ​​generated to express each area to be tested globally. The target adjacency matrix is ​​used to predict the falling objects of each area to be tested, and the global meteorological influence between areas can be considered, so that the falling object prediction results are more accurate, and sanitation workers can be arranged in advance to areas with more falling object predictions, thereby improving the cleaning efficiency of urban falling objects.

[0061] Optionally, in the step of calculating the historical falling object data of each area to be tested based on the historical image data and historical meteorological data corresponding to each area to be tested, the historical image data and the historical meteorological data can be sampled with the same time point as the starting point and through the same time step to obtain N frames of sample image data and corresponding N frames of sample meteorological data, each sample image data corresponds to one sample meteorological data; spatial feature extraction is performed on the falling objects in the n-th frame of sample image data to obtain the spatial features of the falling objects in the n-th frame of sample image data, where n is a positive integer less than or equal to N; aerodynamic feature extraction is performed on the n-th frame of sample meteorological data to obtain the aerodynamic features of the n-th frame of sample meteorological data; feature fusion is performed on the spatial features of the falling objects in the n-th frame of sample image data and the corresponding aerodynamic features of the n-th frame of sample meteorological data to obtain the falling object feature data of the n-th frame; the falling object feature data of N frames are arranged in the time dimension to obtain the historical falling object data of the area to be tested.

[0062] In the embodiment of the present invention, the same time point may be any time point before the current time point, or may be a specified time point. The time step may be an interval time length, for example, a time step of one second indicates an interval time length of one second.

[0063] The historical image data is sampled from a specified time point, with time as the axis and the time step length as the sliding sampling. Each sampling obtains a frame of sample image data, and N samplings are performed to obtain N frames of sample image data. Similarly, the historical meteorological data is collected from a specified time point, with time as the axis and the time step length as the sliding sampling. Each sampling obtains a frame of sample meteorological data, and N samplings are performed to obtain N frames of sample meteorological data. Through sampling, sample image data and sample meteorological data with a smaller data volume can be obtained. Among them, the sample image data corresponds to the sample meteorological data one by one in time.

[0064] The nth frame represents any frame in N frames. The spatial feature extraction of the falling objects in the sample image data of the nth frame is performed to obtain the spatial features of the falling objects in the sample image data of the nth frame. This can mean that the spatial feature extraction of the falling objects in each frame of sample image data is performed to obtain the spatial features of the falling objects in each frame of sample image data. The aerodynamic feature extraction of the sample meteorological data of the nth frame is performed to obtain the aerodynamic features of the sample meteorological data of the nth frame. This can mean that the aerodynamic feature extraction of the falling objects in each frame of sample meteorological data is performed to obtain the aerodynamic features of the sample meteorological data of each frame. The spatial features can be features such as the distribution position of the falling objects, the coverage area of ​​the falling objects, and the area proportion of the falling objects. The aerodynamic features include features such as wind direction, wind force, and air humidity. The spatial features of the falling objects in the sample image data of the nth frame are fused with the aerodynamic features of the corresponding sample meteorological data of the nth frame to obtain the feature data of the falling objects in the nth frame. This can mean that the spatial features of the falling objects in each frame of sample image data are fused with the aerodynamic features of the corresponding frame of sample meteorological data to obtain the feature data of the falling objects in each frame. Specifically, the spatial characteristics and aerodynamic characteristics of the falling objects can be linearly transformed to obtain the transformation characteristics of the same dimension, and the transformation characteristics are channel-fused to obtain the falling object characteristic data. After obtaining the falling object characteristic data of each frame, the falling object characteristic data of all frames are arranged in the time dimension to obtain the historical falling object data of the area to be tested.

[0065] By sampling historical image data and historical meteorological data, sample image data and historical meteorological data are obtained as representatives for subsequent processing, which can reduce the amount of data to be processed. The spatial characteristics of falling objects are extracted through sample image data, and the aerodynamic characteristics are extracted through sample meteorological data, and the spatial characteristics of falling objects are fused with the aerodynamic characteristics, so that the falling object feature data contains both the spatial characteristics of falling objects and the influence of aerodynamics. The falling object feature data of all frames are arranged in the time dimension, and the historical falling object data of the area to be tested are also added with time features, so that the falling object feature data has an implicit change law in the time dimension.

[0066] Optionally, in the step of extracting spatial features of falling objects in the n-th frame sample image data to obtain the spatial features of the falling objects in the n-th frame sample image data, target detection can be performed on the falling objects in the n-th frame sample image data, and spatial distribution features of the falling objects in the n-th frame sample image data can be obtained based on the target detection result; image segmentation can be performed on the falling objects in the n-th frame sample image data, and spatial proportion features of the falling objects in the n-th frame sample image data can be obtained based on the image segmentation result; the spatial distribution features of the falling objects in the n-th frame sample image data and the spatial proportion features of the falling objects in the n-th frame sample image data can be spliced ​​to obtain the spatial features of the falling objects in the n-th frame sample image data.

[0067] In an embodiment of the present invention, falling objects in sample image data can be detected by a target detection algorithm, and the output of the target detection algorithm is a falling object detection frame (x, y, w, h, u), where (x, y) represents the center position of the falling object detection frame, w represents the width of the falling object detection frame, h represents the height of the falling object detection frame, and u represents the confidence of the falling object detection frame. The spatial distribution characteristics of the falling objects can be represented by the falling object detection frame (x, y, w, h, u). The above-mentioned target detection algorithm can be a target detection algorithm such as Faster R-CNN, R-FCN and SSD.

[0068] The falling objects in the sample image data can be segmented by an image segmentation algorithm. Through image segmentation, the mask outline of the falling objects can be obtained. According to the mask outline of the falling objects, the spatial proportion characteristics of the falling objects can be calculated. The above-mentioned spatial proportion characteristics include the coverage area of ​​the falling objects, the area proportion of the falling objects and other characteristics. Among them, the coverage area of ​​the falling objects can be calculated by the mask outline of the falling objects, and the area proportion of the falling objects can be taken as the ratio of the coverage area of ​​the falling objects to the area of ​​the background. The above-mentioned image segmentation algorithm can be a threshold-based segmentation method, an edge-based segmentation method, a region-based segmentation method, a wavelet transform-based segmentation method, and an artificial neural network-based segmentation method.

[0069] After obtaining the spatial distribution characteristics of the falling objects in the sample image data and the spatial proportion characteristics of the falling objects in the sample image data, the spatial distribution characteristics of the falling objects in the sample image data and the spatial proportion characteristics of the falling objects in the sample image data can be spliced ​​together to obtain the spliced ​​characteristics as the spatial characteristics of the falling objects in the sample image data.

[0070] Optionally, in the step of generating a target adjacency matrix based on the positional relationship between each area to be tested and the historical falling object data of each area to be tested, the adjacency relationship between each area to be tested can be determined based on the positional relationship between each area to be tested; based on the adjacency relationship, an initial adjacency matrix between each area to be tested is established; and the historical falling object data of each area to be tested is added to the corresponding matrix unit in the initial adjacency matrix to obtain the target adjacency matrix between each area to be tested.

[0071] In an embodiment of the present invention, each area to be tested is adjacent to at least one other area to be tested, which can be understood as an adjacent relationship between the two areas to be tested or the two areas to be tested are connected through another area to be tested. The positional relationship between the above-mentioned areas to be tested includes an adjacent relationship and an orientation relationship. Each area to be tested can be marked on a map, each area to be tested is used as a matrix unit, two adjacent areas to be tested are used as two adjacent matrix units, two areas to be tested that are adjacent to the left and right are used as two adjacent matrix units to the left and right, and two areas to be tested that are adjacent to each other are used as two adjacent matrix units to the top and bottom. After obtaining the matrix units corresponding to all areas to be tested, determine the number of rows A and the number of columns B of the adjacency matrix to obtain the initial adjacency matrix A*B. Add the historical falling object data of each area to be tested to the corresponding matrix unit in the initial adjacency matrix A*B. For the matrix unit that does not correspond to the area to be tested, fill it with 0 or null value to obtain the target adjacency matrix A*B*N between each area to be tested. The target adjacency matrix includes the spatial dimension A*B and the time dimension N. It should be noted that the number of areas to be tested is less than or equal to A*B.

[0072] Optionally, in the step of predicting falling objects in each area to be tested based on the target adjacency matrix to obtain the falling object prediction results for each area to be tested, global spatial features can be extracted from the target adjacency matrix in the time dimension to obtain global spatial features corresponding to different time points; based on the time dependency between the global spatial features corresponding to different time points, the time features of the global spatial features corresponding to different time points are encoded to obtain global spatiotemporal features; linear regression is performed on the global time features to obtain the falling object prediction results for each area to be tested.

[0073] In an embodiment of the present invention, the target adjacency matrix A*B*N is a three-dimensional matrix. Specifically, the target adjacency matrix A*B*N can be regarded as a continuous frame image, the number of frame images is N, the size is A*B, and each frame image corresponds to a time point. The global spatial features are extracted for the frame images corresponding to different time points to obtain the global spatial features corresponding to the different time points. The above-mentioned global spatial features refer to the spatial distribution between each area to be tested. It can be analyzed from the urban heat island effect that under the same meteorological conditions, the distribution of different areas in the city has different aerodynamic conditions. Therefore, the spatial distribution between each area to be tested will also affect the drift of the falling objects. Therefore, it is necessary to consider the spatial distribution between each area to be tested, that is, it is necessary to consider the above-mentioned global spatial features. Since the local spatial features have been extracted from the historical falling object data, the local spatial features refer to the spatial distribution of the falling objects in the area to be tested. Combining the global spatial features with the local spatial features can more accurately describe the overall distribution of the falling objects.

[0074] The time dependency relationship between the global spatial features corresponding to the above-mentioned different time points can be understood as the temporal correlation relationship between the global spatial features. The global spatial features corresponding to different time points are arranged in chronological order, and the arranged global spatial features are linearly transformed so that the global spatial features corresponding to different time points have time attributes, thereby realizing the encoding of the time features of the global spatial features, and using the global spatial features after linear transformation as the global spatiotemporal features.

[0075] Perform linear regression on the global time features to obtain the prediction results of the falling objects in each area to be tested. The above linear regression uses a fully connected layer algorithm to regress the global time features into a prediction matrix of A*B*T, where A*B is the spatial dimension and T is the prediction time dimension. The matrix unit corresponding to each area to be tested can be found in the spatial dimension of the prediction matrix, and the specific prediction time can be found in the prediction matrix. The prediction value of each area to be tested in the future time can be obtained through the prediction matrix. The prediction value is the spatial distribution of falling objects, such as the distribution position of falling objects, the coverage area of ​​falling objects, the area proportion of falling objects, etc. When the number of distribution positions of a certain area to be tested is greater than the preset number, or when the coverage area of ​​falling objects is greater than the preset area, or when the area proportion of falling objects is greater than the preset proportion, it means that at a certain time in the future, the area to be tested needs to be cleaned, and sanitation workers can be arranged in advance to the area to be tested.

[0076] Optionally, in the step of encoding the time features of the global spatial features corresponding to different time points based on the time dependency between the global spatial features corresponding to different time points to obtain the global spatiotemporal features, the time dimension of the target adjacency matrix can be segmented to obtain K time periods, where N is an integer multiple of K; the global spatial features corresponding to different time points are expanded in one dimension to obtain a global spatial vector; according to the time dependency between the global spatial vectors in each time period, each time period is vector-encoded to obtain K spatiotemporal sub-features; the K spatiotemporal sub-features are clustered by a preset clustering method to obtain the cluster center as the global spatiotemporal feature.

[0077] In an embodiment of the present invention, since the drift of falling objects has a process, in this process, the falling objects at adjacent time points have a strong correlation in spatial distribution. Therefore, the time dimension of the target adjacency matrix can be evenly segmented to obtain K time periods, each time period includes m time points, then N=K*m, each time period includes m global spatial features with time dependence, assuming that the global spatial feature is c*d, then a space-time sub-feature can be expressed as c*d*m, where m is the time dimension of the space-time sub-feature.

[0078] K spatiotemporal sub-features are clustered by a preset clustering method to obtain a cluster center as a global spatiotemporal feature. The preset clustering method may be a K-means clustering algorithm or a similarity clustering algorithm. After clustering is completed, each cluster center is used as a global spatiotemporal feature, and all global spatiotemporal features are subjected to linear regression to obtain the prediction results of falling objects in each test area.

[0079] Optionally, in the step of clustering K spatiotemporal sub-features by a preset clustering method to obtain the cluster center as the global spatiotemporal feature, the path probability from the kth spatiotemporal sub-feature to the jth cluster space can be calculated according to the preset path coefficient; the path result from the kth spatiotemporal sub-feature to the jth cluster space can be calculated according to the preset path coefficient;

[0080] According to the path result from the kth spatiotemporal sub-feature to the jth clustering space, the clustering space parameters of the jth clustering space are calculated as the global spatiotemporal feature.

[0081] In the embodiment of the present invention, the kth spatiotemporal sub-feature can be multiplied by the weight parameter of the jth clustering space to obtain the path result, and the clustering space parameter can be calculated based on the path result and the dynamic path coefficient, and the path result and the path probability are calculated to obtain the path probability, and the spatiotemporal sub-features can be clustered according to the path probability. Specifically, the path probability can be calculated according to the following formula:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] e kj =e kj +d kj

[0088] Among them, the above represents the path result of the kth spatiotemporal sub-feature and the jth clustering space. k represents the kth spatiotemporal sub-feature, the above W kj represents the weight parameter between the kth spatiotemporal sub-feature and the jth clustering space, which can be obtained through an iterative process. The above r represents the number of iterations, and the above c kj represents the dynamic path coefficient, the above e kj Represents the iteration variable. The initial iteration variable can be set to 0. j Indicates the intermediate calculation result. j represents the clustering space parameter, the above d kj Indicates the similarity between the path result and the cluster space parameter. kj The larger it is, the higher the probability that the kth spatiotemporal sub-feature belongs to the jth cluster space.

[0089] The above process is iterated and calculated, and clustering is completed when the specified number of iterations is reached. After clustering is completed, the clustering space parameters corresponding to each clustering space are used as global spatiotemporal features for linear regression.

[0090] It should be noted that the falling object prediction method provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform falling object prediction.

[0091] Optional, see Figure 2 , Figure 2 is a schematic diagram of the structure of a falling object prediction device provided by an embodiment of the present invention, such as Figure 2 As shown, the device comprises:

[0092] An acquisition module 201 is used to acquire historical image data and historical meteorological data of each area to be tested;

[0093] A calculation module 202, configured to calculate the historical falling object data of each of the areas to be measured according to the historical image data and the historical meteorological data corresponding to each of the areas to be measured;

[0094] A generating module 203 is used to generate a target adjacency matrix between each of the areas to be tested according to the positional relationship between each of the areas to be tested and the historical falling object data of each of the areas to be tested;

[0095] The prediction module 204 is used to predict the falling objects in each of the areas to be tested based on the target adjacency matrix to obtain the falling object prediction results of each of the areas to be tested.

[0096] Optionally, the calculation module 202 is also used to sample the historical image data and the historical meteorological data with the same time point as the starting point and through the same time step to obtain N frames of sample image data and corresponding N frames of sample meteorological data, each of the sample image data corresponds to one sample meteorological data; perform spatial feature extraction on the falling objects in the sample image data of the nth frame to obtain the spatial features of the falling objects in the sample image data of the nth frame, where n is a positive integer less than or equal to N; perform aerodynamic feature extraction on the sample meteorological data of the nth frame to obtain the aerodynamic features of the sample meteorological data of the nth frame; perform feature fusion on the spatial features of the falling objects in the sample image data of the nth frame and the corresponding aerodynamic features of the sample meteorological data of the nth frame to obtain the falling object feature data of the nth frame; arrange the falling object feature data of N frames in the time dimension to obtain the historical falling object data of the area to be tested.

[0097] Optionally, the calculation module 202 is further used to perform target detection on the falling objects in the sample image data of the nth frame, and obtain spatial distribution characteristics of the falling objects in the sample image data of the nth frame according to the target detection result; perform image segmentation on the falling objects in the sample image data of the nth frame, and obtain spatial proportion characteristics of the falling objects in the sample image data of the nth frame according to the image segmentation result; and splice the spatial distribution characteristics of the falling objects in the sample image data of the nth frame with the spatial proportion characteristics of the falling objects in the sample image data of the nth frame to obtain the spatial characteristics of the falling objects in the sample image data of the nth frame.

[0098] Optionally, the generation module 203 is also used to determine the adjacency relationship between each of the areas to be tested based on the positional relationship between each of the areas to be tested; establish an initial adjacency matrix between each of the areas to be tested based on the adjacency relationship; add the historical falling object data of each of the areas to be tested to the corresponding matrix cells in the initial adjacency matrix to obtain a target adjacency matrix between each of the areas to be tested.

[0099] Optionally, the prediction module 204 is also used to extract global spatial features of the target adjacency matrix in the time dimension to obtain global spatial features corresponding to different time points; based on the time dependency between the global spatial features corresponding to the different time points, encode the time features of the global spatial features corresponding to the different time points to obtain global spatiotemporal features; perform linear regression on the global time features to obtain the falling object prediction results of each of the tested areas.

[0100] Optionally, the prediction module 204 is also used to segment the time dimension of the target adjacency matrix to obtain K time periods, where N is an integer multiple of K; perform one-dimensional expansion of the global spatial features corresponding to the different time points to obtain a global spatial vector; perform vector encoding on each time period according to the time dependency between the global spatial vectors in each time period to obtain K space-time sub-features; cluster the K space-time sub-features by a preset clustering method to obtain the cluster center as the global space-time feature.

[0101] Optionally, the prediction module 204 is also used to calculate the path result from the kth space-time sub-feature to the jth clustering space according to a preset path coefficient; and based on the path result from the kth space-time sub-feature to the jth clustering space, calculate the clustering space parameters of the jth clustering space as the global space-time feature.

[0102] It should be noted that the falling object prediction device provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform falling object prediction.

[0103] The falling object prediction device provided in the embodiment of the present invention can implement each process implemented by the falling object prediction method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0104] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program of a falling object prediction method stored in the memory 302 and executable on the processor 301, wherein:

[0105] The processor 301 is used to call the computer program stored in the memory 302 and execute the following steps:

[0106] Obtain historical image data and historical meteorological data for each area to be tested;

[0107] Calculate the historical falling object data of each of the areas to be tested according to the historical image data and the historical meteorological data corresponding to each of the areas to be tested;

[0108] Generate a target adjacency matrix between the areas to be tested according to the positional relationship between the areas to be tested and the historical falling object data of the areas to be tested;

[0109] Based on the target adjacency matrix, the falling objects in each of the areas to be tested are predicted to obtain the falling object prediction results for each of the areas to be tested.

[0110] Optionally, the processor 301 calculates the historical falling object data of each of the areas to be tested according to the historical image data and the historical meteorological data corresponding to each of the areas to be tested, including:

[0111] The historical image data and the historical meteorological data are sampled with the same time point as the starting point and the same time step to obtain N frames of sample image data and corresponding N frames of sample meteorological data, each of the sample image data corresponds to one frame of sample meteorological data;

[0112] Extracting spatial features of the falling objects in the sample image data of the nth frame to obtain spatial features of the falling objects in the sample image data of the nth frame, where n is a positive integer less than or equal to N;

[0113] Extracting aerodynamic features of the sample meteorological data of the nth frame to obtain aerodynamic features of the sample meteorological data of the nth frame;

[0114] The spatial features of the falling objects in the sample image data of the nth frame are combined with the aerodynamic features of the corresponding sample meteorological data of the nth frame to obtain the falling object feature data of the nth frame;

[0115] Arrange the N frames of the falling object feature data in the time dimension to obtain the historical falling object data of the area to be tested.

[0116] Optionally, the processor 301 performs the extraction of spatial features of the falling objects in the sample image data of the nth frame to obtain the spatial features of the falling objects in the sample image data of the nth frame, including:

[0117] Performing target detection on the falling objects in the sample image data of the nth frame, and obtaining the spatial distribution characteristics of the falling objects in the sample image data of the nth frame according to the target detection result;

[0118] Performing image segmentation on the falling objects in the sample image data of the nth frame, and obtaining the spatial proportion characteristics of the falling objects in the sample image data of the nth frame according to the image segmentation result;

[0119] The spatial distribution characteristics of the falling objects in the sample image data of the nth frame are spliced ​​with the spatial proportion characteristics of the falling objects in the sample image data of the nth frame to obtain the spatial characteristics of the falling objects in the sample image data of the nth frame.

[0120] Optionally, the processor 301 generates a target adjacency matrix according to the positional relationship between the areas to be detected and the historical falling object data of the areas to be detected, including:

[0121] Determining the adjacency relationship of each of the areas to be tested according to the positional relationship of each of the areas to be tested;

[0122] According to the adjacency relationship, an initial adjacency matrix between the regions to be tested is established;

[0123] The historical falling object data of each of the areas to be tested is added to the corresponding matrix unit in the initial adjacency matrix to obtain the target adjacency matrix between each of the areas to be tested.

[0124] Optionally, the processor 301 performs the prediction of falling objects for each of the areas to be tested based on the target adjacency matrix to obtain the falling object prediction results for each of the areas to be tested, including:

[0125] Extracting global spatial features of the target adjacency matrix in the time dimension to obtain global spatial features corresponding to different time points;

[0126] Based on the time dependency relationship between the global spatial features corresponding to the different time points, the time features of the global spatial features corresponding to the different time points are encoded to obtain global spatiotemporal features;

[0127] Linear regression is performed on the global time feature to obtain the falling object prediction results of each of the areas to be tested.

[0128] Optionally, the encoding of the time features of the global spatial features corresponding to the different time points based on the time dependency between the global spatial features corresponding to the different time points by the processor 301 to obtain the global spatiotemporal features includes:

[0129] Segmenting the time dimension of the target adjacency matrix to obtain K time periods, where N is an integer multiple of K;

[0130] Expand the global space features corresponding to the different time points in one dimension to obtain a global space vector;

[0131] According to the time dependency between the global space vectors in each time period, vector encoding is performed on each time period to obtain K spatiotemporal sub-features;

[0132] The K spatiotemporal sub-features are clustered by a preset clustering method to obtain a cluster center as a global spatiotemporal feature.

[0133] Optionally, the processor 301 performs clustering of the K spatiotemporal sub-features by a preset clustering method to obtain a cluster center as a global spatiotemporal feature, including:

[0134] According to the preset path coefficient, the path result from the kth spatiotemporal sub-feature to the jth cluster space is calculated;

[0135] According to the path result from the kth spatiotemporal sub-feature to the jth clustering space, the clustering space parameters of the jth clustering space are calculated as the global spatiotemporal feature.

[0136] The electronic device provided in the embodiment of the present invention can implement each process implemented by the falling object prediction method in the above method embodiment and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0137] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the falling object prediction method provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0138] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only storage memory (RON) or a random access memory (RAN).

[0139] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A falling object prediction method, characterized in that: The following steps are involved: Obtain historical image data and historical meteorological data for each area to be tested; The historical image data and the historical meteorological data are sampled with the same time point as the starting point and the same time step to obtain N frames of sample image data and corresponding N frames of sample meteorological data, each of the sample image data corresponds to one frame of sample meteorological data; Extracting spatial features of the falling objects in the sample image data of the nth frame to obtain spatial features of the falling objects in the sample image data of the nth frame, where n is a positive integer less than or equal to N; Extracting aerodynamic features of the sample meteorological data of the nth frame to obtain aerodynamic features of the sample meteorological data of the nth frame; The spatial features of the falling objects in the sample image data of the nth frame are combined with the aerodynamic features of the corresponding sample meteorological data of the nth frame to obtain the falling object feature data of the nth frame; Arrange the N frames of the falling object feature data in the time dimension to obtain the historical falling object data of the area to be tested; Generate a target adjacency matrix between the areas to be tested according to the positional relationship between the areas to be tested and the historical falling object data of the areas to be tested; the target adjacency matrix includes a spatial dimension and a temporal dimension, the spatial dimension is determined according to the number of areas to be tested, and the temporal dimension is determined according to the historical moments contained in the historical falling object data; Extracting global spatial features of the target adjacency matrix in the time dimension to obtain global spatial features corresponding to different time points; Segmenting the time dimension of the target adjacency matrix to obtain K time periods, where N is an integer multiple of K; Expand the global space features corresponding to the different time points in one dimension to obtain a global space vector; According to the time dependency between the global space vectors in each time period, vector encoding is performed on each time period to obtain K spatiotemporal sub-features; According to the preset path coefficient, the path result from the kth spatiotemporal sub-feature to the jth cluster space is calculated; According to the path result from the kth spatiotemporal sub-feature to the jth clustering space, the clustering space parameter of the jth clustering space is calculated as the global spatiotemporal feature; Linear regression is performed on the global spatiotemporal features to obtain the falling object prediction results of each of the areas to be tested.

2. The falling object prediction method according to claim 1, characterized in that: The extracting spatial features of the falling objects in the sample image data of the nth frame to obtain the spatial features of the falling objects in the sample image data of the nth frame includes: Performing target detection on the falling objects in the sample image data of the nth frame, and obtaining the spatial distribution characteristics of the falling objects in the sample image data of the nth frame according to the target detection result; Performing image segmentation on the falling objects in the sample image data of the nth frame, and obtaining the spatial proportion characteristics of the falling objects in the sample image data of the nth frame according to the image segmentation result; The spatial distribution characteristics of the falling objects in the sample image data of the nth frame are spliced ​​with the spatial proportion characteristics of the falling objects in the sample image data of the nth frame to obtain the spatial characteristics of the falling objects in the sample image data of the nth frame.

3. The falling object prediction method according to claim 2, characterized in that: The generating a target adjacency matrix according to the positional relationship between the areas to be tested and the historical falling object data of the areas to be tested comprises: Determining the adjacency relationship of each of the areas to be tested according to the positional relationship of each of the areas to be tested; According to the adjacency relationship, an initial adjacency matrix between the regions to be tested is established; The historical falling object data of each of the areas to be tested is added to the corresponding matrix unit in the initial adjacency matrix to obtain the target adjacency matrix between each of the areas to be tested.

4. A falling object prediction device, characterized in that: The device comprises: An acquisition module is used to acquire historical image data and historical meteorological data of each area to be tested; A calculation module is used to sample the historical image data and the historical meteorological data with the same time point as the starting point and the same time step to obtain N frames of sample image data and corresponding N frames of sample meteorological data, each of which corresponds to one sample meteorological data; extract spatial features of the falling objects in the sample image data of the nth frame to obtain the spatial features of the falling objects in the sample image data of the nth frame, where n is a positive integer less than or equal to N; extract aerodynamic features of the sample meteorological data of the nth frame to obtain the aerodynamic features of the sample meteorological data of the nth frame; perform feature fusion on the spatial features of the falling objects in the sample image data of the nth frame and the aerodynamic features of the corresponding sample meteorological data of the nth frame to obtain the falling object feature data of the nth frame; arrange the falling object feature data of N frames in the time dimension to obtain the historical falling object data of the area to be measured; A generation module is used to generate a target adjacency matrix between each of the areas to be tested according to the positional relationship between each of the areas to be tested and the historical falling object data of each of the areas to be tested; the target adjacency matrix includes a spatial dimension and a temporal dimension, the spatial dimension is determined according to the number of areas to be tested, and the temporal dimension is determined according to the historical moments contained in the historical falling object data; The prediction module is used to extract global spatial features of the target adjacency matrix in the time dimension to obtain global spatial features corresponding to different time points; segment the time dimension of the target adjacency matrix to obtain K time periods, where N is an integer multiple of K; perform one-dimensional expansion of the global spatial features corresponding to the different time points to obtain global spatial vectors; perform vector encoding on each time period according to the time dependency between the global spatial vectors in each time period to obtain K spatiotemporal sub-features; calculate the path result from the kth spatiotemporal sub-feature to the jth clustering space according to a preset path coefficient; calculate the clustering space parameters of the jth clustering space as the global spatiotemporal features according to the path result from the kth spatiotemporal sub-feature to the jth clustering space; perform linear regression on the global spatiotemporal features to obtain the falling object prediction results of each of the test areas.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the falling object prediction method as claimed in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the falling object prediction method according to any one of claims 1 to 3 are implemented.

Images