Image processing method, computer readable storage medium, and computer terminal

Abstract

This application discloses an image processing method, a computer-readable storage medium, and a computer terminal. The method comprises: obtaining a historical image sequence, wherein the historical image sequence is a sequence of images collected within a first time period; and using an image prediction model to predict the historical image sequence based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, the second time period being a period after the first time period. This application solves the technical problem in related arts of the difficulty in performing long-term predictions for changing scenes.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, a computer-readable storage medium, and a computer terminal. Background Art

[0002] Currently, short-term extrapolation is commonly used to predict changing scenarios. For example, in meteorological application scenarios, short-term extrapolation is used based on meteorological images to forecast precipitation in future time periods. However, when using traditional short-term extrapolation methods to predict changing scenarios, the prediction time and video memory consumption will increase linearly with the increase in prediction time, which makes it difficult to meet the needs of large-scale and long-term predictions, resulting in a limited prediction range.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, a computer-readable storage medium, and a computer terminal to at least solve the technical problem in the related art that it is difficult to make long-term predictions for changing scenes.

[0005] According to one aspect of an embodiment of the present application, an image processing method is provided, including: obtaining a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; using an image prediction model to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence, to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, and the second time period is a time period after the first time period.

[0006] According to one aspect of an embodiment of the present application, an image processing method is provided, including: obtaining a historical meteorological map sequence, wherein the historical meteorological map sequence is a meteorological map sequence collected within a first time period; using a meteorological prediction model to perform prediction processing on the historical meteorological map sequence based on a first feature corresponding to the historical meteorological map sequence, to obtain a target meteorological map sequence within a second time period, wherein the first feature is used to characterize the motion state of the meteorological target contained in the historical meteorological map sequence within the first time period and a second time period, and the second time period is a time period after the first time period.

[0007] According to one aspect of an embodiment of the present application, an image processing method is provided, including: a cloud server obtains a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; the cloud server uses an image prediction model to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence, to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, and the second time period is a time period after the first time period.

[0008] According to another aspect of an embodiment of the present application, another image processing device is provided, including: a first acquisition module, used to acquire a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; a first prediction module, used to use an image prediction model to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence, to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, and the second time period is a time period after the first time period.

[0009] According to another aspect of an embodiment of the present application, another image processing device is provided, including: a second acquisition module, used to acquire a historical meteorological map sequence, wherein the historical meteorological map sequence is a meteorological map sequence collected within a first time period; a second prediction module, used to use a meteorological prediction model to perform prediction processing on the historical meteorological map sequence based on a first feature corresponding to the historical meteorological map sequence, to obtain a target meteorological map sequence within a second time period, wherein the first feature is used to characterize the motion state of the meteorological target contained in the historical meteorological map sequence within the first time period and a second time period, and the second time period is the time period after the first time period.

[0010] According to another aspect of an embodiment of the present application, another image processing device is provided, including: a third acquisition module for acquiring a historical image sequence through a cloud server, wherein the historical image sequence is an image sequence collected within a first time period; a third prediction module for using an image prediction model through a cloud server to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence, to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, and the second time period is a time period after the first time period.

[0011] According to another aspect of the embodiments of the present application, a computer-readable storage medium is also provided, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the image processing method in any of the above embodiments.

[0012] According to another aspect of the embodiments of the present application, a computer terminal is further provided, including: a processor and a memory, wherein the processor is configured to run a program stored in the memory, wherein the image processing method in any one of the above embodiments is executed when the program is run.

[0013] In an embodiment of the present application, a historical image sequence is first acquired, wherein the historical image sequence is an image sequence collected within a first time period. Then, an image prediction model is used to predict the historical image sequence based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and the second time period, and the second time period is the time period after the first time period, thereby achieving the purpose of predicting the target image sequence. It is easy to notice that since the first feature corresponding to the historical image sequence contains the motion state of the target within the first time period and the second time period, the first feature is the long-term motion feature of the target. Since the image prediction model uses the long-term motion feature as an auxiliary feature to predict the historical image sequence, the accuracy of the target image sequence predicted over a long period of time can be ensured, thereby achieving the technical effect of improving the accuracy of the prediction of the historical image sequence over a long period of time, thereby solving the technical problem in the related art that it is difficult to perform long-term predictions for changing scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method according to an embodiment of the present application;

[0016] Figure 2 is a flowchart of an image processing method according to Example 1 of the present application;

[0017] Figure 3 is a schematic diagram of an interactive interface according to an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of an optional interactive interface according to an embodiment of the present application;

[0019] Figure 5 is a flowchart of another image processing method according to Example 1 of the present application;

[0020] Figure 6is a schematic diagram of an image processing process according to an embodiment of the present application;

[0021] Figure 7 is a flowchart of an image processing method according to Example 2 of the present application;

[0022] Figure 8 is a flowchart of an image processing method according to Example 3 of the present application;

[0023] Figure 9 is a schematic diagram of an image processing device according to Embodiment 4 of the present application;

[0024] Figure 10 is a schematic diagram of an image processing device according to Embodiment 5 of the present application;

[0025] Figure 11 is a schematic diagram of an image processing device according to Example 6 of the present application;

[0026] Figure 12 A structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0030] Radar extrapolation: It can be the use of radar echo sequences to predict future radar echo sequences.

[0031] Memory bank: can be a memory module or a memory bank, M={mi} can be in matrix form, used to store long-term motion information.

[0032] Current radar short-term extrapolation schemes only use historical radar echo motion trend information to extrapolate future motion states. This lack of awareness of long-term motion limits their effectiveness. LMC-Memory (LMC-Memory) utilizes a memory bank mechanism for video prediction. However, this scheme is based on an LSTM (Lon Short-Term Memory) recurrent neural network model. Its drawback is that both prediction time and video memory usage increase linearly with the prediction duration, making it difficult to predict large, long-term sequences.

[0033] In order to solve the above problems, the present application provides an image processing method, which can extend the predicted time range and improve the accuracy of predicting long time ranges by fusing the extracted second features with the long time series features predicted in advance and making predictions based on the fused features.

[0034] Example 1

[0035] According to an embodiment of the present application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned image processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0041] Figure 1The hardware structure block diagram shown can be used not only as an exemplary block diagram of the computer terminal 10 (or mobile device), but also as an exemplary block diagram of the server. In an optional embodiment, the computer terminal 10 (or mobile device) can be a locally deployed server or a cloud server, connected to one or more clients via a data network connection or electronic connection. The data network connection can be a local area network connection, a wide area network connection, an Internet connection, or other types of data network connections.

[0042] Under the above operating environment, this application provides Figure 2 The image processing method shown. Figure 2 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0043] Step S202: Acquire a historical image sequence.

[0044] The historical image sequence is an image sequence collected within the first time period.

[0045] The first time period mentioned above can be set according to needs.

[0046] The aforementioned image sequence may be an image sequence arranged by time. The aforementioned historical image sequence may be an image sequence arranged by time within a historical time period. The historical image sequence may include multiple consecutive historical images, each of which may be separated by a first time period, which can be set arbitrarily. Optionally, the images in the historical image sequence may be remote sensing images, aerial images taken by drones, radar images, or images taken by satellites, but are not limited thereto. In different application scenarios, the types of objects contained in each historical image sequence may vary. For example, in meteorological application scenarios, the objects contained in the historical image sequence may include tornadoes, clouds, rain, snow, hail, typhoons, etc.; in agriculture and forestry application scenarios, the objects contained in the historical image sequence may include crops such as wheat, rice, and potatoes; in natural resources application scenarios, the objects contained in the historical image sequence may include rivers and lava; in urban planning application scenarios, the objects contained in the historical image sequence may include buildings under construction; and in transportation application scenarios, the objects contained in the historical image sequence may include vehicles, subways, airplanes, ships, etc.

[0047] In an optional embodiment, the historical image sequence may be an image sequence taken by a satellite or drone before the current moment. The satellite or drone may continuously take multiple images as an image sequence. When it is necessary to analyze or process the previously taken image sequence, the historical image sequence may be searched according to the start time point and the end time point of the first time period and transmitted to the server via the network for processing by the server. At the same time, the historical image sequence may be displayed to the user, such as Figure 3 As shown, the historical image sequence can be displayed in the image acquisition area; in another optional embodiment, the historical image sequence can be taken by a satellite or a drone and uploaded to the server by the user, and processed by the server, such as Figure 4 As shown, users can upload historical image sequences to the server by clicking the "Upload Image" button in the interactive interface or by directly dragging historical image sequences into the dotted box. The uploaded historical image sequences can then be displayed within the image acquisition area. The server here can be deployed locally or in the cloud.

[0048] Step S204 : using the image prediction model to perform prediction processing on the historical image sequence based on the first feature corresponding to the historical image sequence, to obtain a target image sequence within the second time period.

[0049] The first feature is used to characterize the motion state of the target contained in the historical image sequence within a first time period and a second time period, and the second time period is a time period after the first time period.

[0050] The first time period mentioned above may be the entire time period of the historical image sequence, may be a partial time period of the historical image sequence, or may be a designated time period.

[0051] In an optional embodiment, the historical image sequence can be encoded using an encoder of a u2net network (Convolutional Networks for Biomedical Image Segmentation, a deep learning-based image semantic segmentation method) to obtain image features, and then the second features of the target within the first time period can be determined from the image features.

[0052] In another optional embodiment, the second feature can be fused with the first feature to obtain a fused feature, which can then be used to predict the target image sequence within the second time period using an image prediction model. By using the first feature as an auxiliary feature in combination with the second feature to predict the historical image sequence, the accuracy of the prediction can be improved, thereby improving the accuracy of the target image sequence.

[0053] In another optional embodiment, a C3D encoder can be used to encode a historical image sequence, and a 3D convolutional neural network can be used to obtain a third feature of the target in the historical image sequence in space. After obtaining the third feature, the motion state of the target can be reflected according to the spatial vector of the third feature, and the vector corresponding to the target can be retrieved from the memory bank based on the spatial vector, and the long-term time series feature corresponding to the vector can be obtained as the first feature.

[0054] The above 3D convolutional neural network mainly uses 3D convolution to extract features in three-dimensional space.

[0055] In another optional embodiment, after obtaining the spatial vector corresponding to the third feature of the target, the spatial vector can be compared with the vectors in the image sequence pre-stored in the memory bank to determine the vector with higher similarity, and the first feature can be determined based on the vector with higher similarity.

[0056] In another optional embodiment, since the first feature includes the motion state of the target in the first time period and the predicted motion state of the target in the second time period, the first feature can be used as an auxiliary feature of the second feature to predict the target image sequence in the second time period, which can improve the prediction accuracy.

[0057] In another optional embodiment, the historical image sequence can be predicted to obtain a second feature of the target in the first time period. While obtaining the second feature, the historical image sequence can be encoded using a C3D encoder, and a 3D convolutional neural network can be used to obtain a third feature of the historical image sequence in three-dimensional space. The third feature can be used to retrieve a long-term feature corresponding to the historical image sequence in the memory bank, that is, the first feature including the first time period and the second time period. By using the retrieved first feature as an auxiliary feature combined with the second feature to predict the historical image sequence, the image sequence of the target in a longer time period can be predicted more accurately. Therefore, compared with the method of directly predicting based on the historical image sequence of the target, it has a longer prediction range and higher accuracy.

[0058] For example, in a meteorological application scenario, a historical image sequence of a tornado can be predicted and processed. While obtaining the motion changes and morphological changes of the tornado in a first time period, that is, obtaining the second feature of the tornado, the C3D encoder can be used to obtain the historical image sequence for encoding processing, and the 3D convolutional neural network can be used to obtain the third feature of the historical image sequence in three-dimensional space. The third feature is used to retrieve the long-term time series feature corresponding to the historical image sequence in the memory bank, that is, the first feature. By using the retrieved first feature as an auxiliary feature combined with the second feature to predict the historical image sequence, the motion changes and morphological changes of the tornado in a longer time period can be more accurately predicted. Therefore, compared with the method of directly predicting based on the historical image sequence of the tornado, it has a longer prediction range and higher accuracy.

[0059] For another example, in a meteorological application scenario, a historical image sequence of rainfall can be predicted and processed. While obtaining the increasing or decreasing rainfall state in the first time period, that is, obtaining the second feature of the rainfall, the historical image sequence can be obtained using a C3D encoder for encoding processing, and the third feature of the historical image sequence in three-dimensional space can be obtained using a 3D convolutional neural network. The third feature is used to retrieve the long-term time series feature corresponding to the historical image sequence in the memory bank, that is, the first feature. By using the retrieved first feature as an auxiliary feature combined with the second feature to predict the historical image sequence, the increasing or decreasing rainfall state in a longer time period can be more accurately predicted. Therefore, compared with the method of directly predicting based on the historical image sequence of rainfall, it has a longer prediction range and higher accuracy.

[0060] For another example, in agricultural and forestry application scenarios, the historical image sequence of wheat can be predicted and processed. While obtaining the growth of wheat in the first time period, that is, obtaining the second feature of wheat, the C3D encoder can be used to obtain the historical image sequence for encoding processing, and the 3D convolutional neural network can be used to obtain the third feature of the historical image sequence in three-dimensional space. The third feature is used to retrieve the long-term time series feature corresponding to the historical image sequence in the memory bank, that is, the first feature. By using the retrieved first feature as an auxiliary feature combined with the second feature to predict the historical image sequence, the growth of wheat in a longer time period can be predicted more accurately. Therefore, compared with the method of directly predicting based on the historical image sequence of wheat, it has a longer prediction range and higher accuracy.

[0061] For another example, in a natural resource application scenario, a historical image sequence of a river can be predicted. While obtaining the flow velocity and direction of the river in a first time period, that is, obtaining the second feature of the river, a C3D encoder can be used to obtain the historical image sequence for encoding processing, and a 3D convolutional neural network can be used to obtain the third feature of the historical image sequence in three-dimensional space. The third feature is used to retrieve the long-term time series feature corresponding to the historical image sequence in the memory bank, that is, the first feature. By using the retrieved first feature as an auxiliary feature combined with the second feature to predict the historical image sequence, the flow velocity and direction of the river in a longer time period can be predicted more accurately. Therefore, compared with the method of directly predicting based on the historical image sequence of the river, it has a longer prediction range and higher accuracy.

[0062] In another optional embodiment, after the server performs predictive processing on the second feature and the first feature to obtain the target image sequence within the second time period, the server can directly display it to the user for viewing; in another optional embodiment, after the server performs predictive processing on the second feature and the first feature to obtain the target image sequence within the second time period, the server can feed back the target image sequence to the user's client via the network, and the client can display it to the user for viewing. Furthermore, after the target image sequence is displayed to the user, the user can verify whether the target image sequence is correct based on the image sequence obtained in real time within the second time period. If it is not correct, the correct image sequence can be fed back and uploaded to the server, so that the server can train the image prediction model in the prediction process based on the user feedback, thereby improving the server performance.

[0063] For example, using clouds in a meteorological application scenario as an example, after acquiring a historical image sequence of clouds from a satellite, drone, or radar, the image sequence can be sent directly to the server for prediction processing, or it can be transmitted to the client, where the user selects and uploads the image sequence to be predicted. After acquiring the historical image sequence of clouds, the server can perform feature extraction on the image sequence to obtain a second feature corresponding to the historical image sequence, and also obtain a first feature corresponding to the historical image sequence of clouds. The first feature includes the motion state of the clouds during a first time period and the pre-predicted motion state of the clouds during a second time period. After obtaining the second and first features, the server can directly display them to the user, or the server can send them to the client, which then displays them to the user. This allows the user to see both the second and first features, and the prediction process can be optimized based on user feedback, improving server performance.

[0064] Furthermore, after determining the second feature and the first feature, the server performs predictive processing on the historical image sequence based on the second feature and the first feature of the cloud layer to obtain a target image sequence within a second time period. After obtaining the target image sequence, the server can directly display the target image sequence to the user for viewing, or the server sends the target image sequence to the client, which displays it to the user for viewing, so that the user can see the target image sequence, and the prediction processing process can be optimized through user feedback results to further improve the performance of the server.

[0065] Through the solution provided by the above-mentioned embodiment of the present application, a historical image sequence is first obtained, wherein the historical image sequence is an image sequence collected within a first time period. Then, an image prediction model is used to predict the historical image sequence based on the first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and the second time period, and the second time period is the time period after the first time period, thereby achieving the purpose of predicting the target image sequence. It is easy to notice that since the first feature corresponding to the historical image sequence contains the motion state of the target within the first time period and the second time period, the first feature is the long-term motion feature of the target. Since the image prediction model uses the long-term motion feature as an auxiliary feature to predict the historical image sequence, the accuracy of the target image sequence obtained by long-term prediction can be ensured, so as to achieve the technical effect of improving the prediction accuracy of the historical image sequence over a long period of time, thereby solving the technical problem in the related art that it is difficult to perform long-term prediction for changing scenes.

[0066] Optionally, the image prediction model includes: a prediction network and a memory module, wherein the image prediction model is used to perform prediction processing on the historical image sequence based on the first feature corresponding to the historical image sequence to obtain the target image sequence in the second time period, which includes: using the prediction network to extract features of the historical image sequence to obtain the second feature corresponding to the historical image sequence, wherein the second feature is used to characterize the motion state of the target in the first time period; obtaining the first feature stored in the memory module; and using the prediction network to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0067] The above-mentioned prediction network may be a feature extraction network for extracting features from historical image sequences.

[0068] The above-mentioned memory module may be a storage space specifically used to store features corresponding to an image sequence, for example, a memory or a memory bank. The present application takes a memory bank as an example for explanation.

[0069] In an optional embodiment, a prediction network can be used to extract features from a historical image sequence to obtain a second feature corresponding to the historical image sequence. After obtaining the second feature, the second feature can be used to obtain a long-term feature corresponding to the second feature in a memory module, that is, a first feature. After obtaining the first feature, the first feature and the second feature can be fused to obtain a fused feature, and the fused feature can be predicted and processed using a prediction network to obtain a target image sequence.

[0070] In the above embodiment of the present application, the image prediction model also includes: a feature extraction network, and obtaining the first feature stored in the memory module includes: using the feature extraction network to process the historical image sequence to obtain a third feature corresponding to the historical image sequence; obtaining the first vector corresponding to each spatial position in the third feature; and obtaining the first feature based on the first vector corresponding to each spatial position and multiple second vectors stored in the memory module.

[0071] The above-mentioned feature extraction network can be a three-dimensional convolutional network, which can be composed of a C3D encoder and a 3D convolutional neural network, wherein the C3D encoder is used to encode the features of the targets in the historical image sequence in three-dimensional space, and the 3D convolutional neural network is used to extract the encoded features.

[0072] In an optional embodiment, the memory module may be a memory bank, which is used to store the second vectors corresponding to the historical image sequence and the predicted target image sequence, wherein the second vectors are presented in the form of a matrix, each row in the matrix is ​​a second vector, and each second vector represents a motion state.

[0073] In an optional embodiment, a 3D convolutional neural network can be used to perform feature extraction on the historical image sequence to obtain abstract features of the historical image sequence in three-dimensional space, and obtain the first vector corresponding to each spatial position in the third feature. After obtaining the first vector, the second vector corresponding to the spatial position can be found in the storage based on the spatial position of the first vector, and the first feature can be obtained based on the similarity between the first vector and the second vector.

[0074] In the above embodiment of the present application, obtaining the first feature based on the first vector corresponding to each spatial position and the multiple second vectors stored in the memory module includes: obtaining the similarity between the first vector and each second vector to obtain multiple similarities; obtaining the weighted sum of each similarity and the corresponding second vector to obtain the time series feature corresponding to each spatial position; and obtaining the first feature based on the time series features corresponding to all spatial positions in the third feature.

[0075] The above-mentioned time series features may be time series features, which are used to reflect the changing state of the target.

[0076] In an optional embodiment, the similarity between the first vector and each second vector can be obtained by using the cosine function to obtain multiple similarities, and the time series features corresponding to each spatial position can be obtained based on the weighted sum of the multiple similarities and the second vectors. In this way, the time series features corresponding to all spatial positions in the third feature can be determined. After obtaining all the time series features, the time series features of all positions can be combined according to the position to obtain the first feature.

[0077] In another optional embodiment, multiple similarities can be obtained by the following formula:

[0078]

[0079] Among them, z i，j is the first vector, m l is the second vector, d() is the cosine function, w l For similarity.

[0080] In another optional embodiment, the weighted sum of each similarity and the corresponding second vector may be obtained by the following formula:

[0081]

[0082] Among them, w l is the similarity, m l is the second vector, w l m l is the time series feature, f i，j The first feature.

[0083] In the above embodiment of the present application, the encoder of the image prediction model is used to extract features of the historical image sequence to obtain the second feature; the decoder of the image prediction model is used to predict the second feature and the first feature to obtain the target image sequence.

[0084] In an optional embodiment, the encoder of the image prediction model may be used to perform feature encoding processing on the historical image sequence to obtain image features of the historical image sequence, and the second features of the first time period may be determined from the image features of the historical image sequence.

[0085] In the above embodiment of the present application, the method also includes: obtaining a training sample, wherein the training sample includes: a first image sequence and a second image sequence, the first image sequence is an image sequence collected within a third time period, and the second image sequence is an image sequence collected within a fourth time period after the third time period; performing feature extraction on the first image sequence to obtain a first feature corresponding to the training sample, wherein the first feature is used to characterize the motion state of the training target contained in the training sample within the third time period; obtaining a fourth feature corresponding to the training sample, wherein the fourth feature is used to characterize the motion state of the training target within the third time period and the fourth time period; performing predictive processing on the first feature and the fourth feature to obtain a third image sequence within the fourth time period; and updating the parameters of the memory module based on the second image sequence and the third image sequence.

[0086] The first image sequence may be the motion state of the training target in the third time period, and the second image sequence may be an image sequence obtained by predicting the first image sequence.

[0087] In an optional embodiment, the first image sequence may be encoded by an encoder of a u2net network to obtain image features, and then the first features of the training target in the third time period may be determined from the image features.

[0088] In another optional embodiment, the first image sequence and the second image sequence can be spliced ​​together, and the spliced ​​image sequence can be used to retrieve the long-term temporal features of the training target in the memory bank, that is, the fourth feature mentioned above. After obtaining the fourth feature, the fourth feature can be used as an auxiliary feature in combination with the first feature to perform prediction processing on the first image sequence to obtain a third image sequence within a fourth time period, and the parameters of the memory bank can be updated based on the second image sequence and the third image sequence. By updating the parameters in the memory bank, the motion information of the long sequence of the training target can be saved for subsequent retrieval in the actual prediction process.

[0089] In another optional embodiment, a loss function may be calculated based on the second image sequence and the third image sequence, and the parameters in the memory module may be updated based on the loss function.

[0090] In the above embodiment of the present application, the prediction network includes: an encoder composed of multiple encoding layers, and a decoder composed of multiple decoding layers, the output of the memory module is connected to the first decoding layer of the decoder, and the multiple encoding layers are connected to the multiple decoding layers in a one-to-one correspondence, wherein the encoder is used to extract features from the historical image sequence to obtain the second feature, and the decoder is used to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0091] The aforementioned multiple encoding layers may be connected in series, and the aforementioned multiple decoding layers may be connected in series.

[0092] In an optional embodiment, the historical image sequence can be input into the encoder of the prediction network, the historical image sequence can be encoded through multiple encoding layers in the encoder, and then the second feature of the historical image sequence can be output through the last encoding layer. The first feature can be obtained from the memory module, and the first feature and the second feature can be input into the first layer of the decoder, the first feature and the second feature can be predicted through multiple decoders, and the target image sequence can be output through the last decoding layer of the multiple decoding layers.

[0093] In the above embodiment of the present application, the target image sequence is a weather map sequence. After using the image prediction model to predict the historical image sequence based on the first feature corresponding to the historical image sequence to obtain the target image sequence in the second time period, the method also includes: determining the weather information in the second time period based on the target image sequence; and displaying the weather information in the interactive interface.

[0094] The above interactive interface can be as follows Figure 3 The interface shown is used by the user to view the feedback information corresponding to the target image sequence, and to adjust the above results on the interface and feed them back to the server.

[0095] In order to ensure the accuracy of the target image sequence, in an optional embodiment, the server can directly display the target image sequence to the user for viewing, that is, display the target image sequence on the interactive interface. In another optional embodiment, the server can send the target image sequence to the client via the network, and the client can display it to the user for viewing, that is, display the target image sequence on the interactive interface. Furthermore, the target image sequence can be confirmed by the user. If the confirmation is correct, the target image sequence can be directly determined. If the user confirms that it is incorrect, the user can modify the target image sequence on the interactive interface, obtain corresponding feedback information, and feed the feedback information back to the server. The server can process the target image sequence based on the feedback information, and the image prediction model in the prediction process can be trained based on the feedback information to improve the performance of the server.

[0096] In an optional embodiment, when the target image sequence is a meteorological image sequence, meteorological information for a second time period can be determined based on the meteorological image sequence to determine rainfall or other weather conditions for a future time period. The meteorological information can be displayed in an interactive interface to inform the user of the rainfall or other weather conditions for the future time period and remind the user to prepare. Travel information can also be displayed in the interactive interface to prompt the user to choose a suitable means of transportation for travel. Furthermore, the rainfall or other weather conditions for the future time period can be announced in the form of voice broadcasts. The travel information can also be announced in the form of voice broadcasts, and travel recommendations can be announced to the user. Furthermore, the rainfall or other weather conditions for the future time period can be notified to the user in the form of text message reminders to remind the user to prepare. The travel information can also be notified to the user in the form of text message reminders, and travel recommendations can be announced to the user.

[0097] In another optional embodiment, after determining the meteorological information in the second time period based on the target image sequence, it is also possible to determine whether the rainfall exceeds the preset rainfall based on the meteorological information in the second time period. If it exceeds the preset rainfall, it means that the rainfall is large. At this time, a flood prevention and control warning can be issued to remind the public to take preventive measures.

[0098] In the above embodiment of the present application, after determining the meteorological information within the second time period based on the target image sequence, the method further includes: generating travel recommendation information within the second time period based on the meteorological information, wherein the travel recommendation information includes at least one of the following: travel route, travel time, and items to be carried; and displaying the travel recommendation information in the interactive interface.

[0099] In an optional embodiment, if the weather information during the second time period indicates continuous rainfall, travel recommendation information may be generated based on the continuous rainfall weather information. The travel recommendation information may include travel routes with less congestion, travel times when rainfall is relatively light, and umbrellas and raincoats to carry in case of rain. The travel recommendation information may be displayed in the interactive interface to prompt the user to view it, may be announced via voice broadcast, or may be sent to the user's mobile terminal via text message.

[0100] In the above embodiment of the present application, after determining the meteorological information within the second time period based on the target image sequence, the method also includes: obtaining travel information, wherein the travel information is determined based on the travel schedule or feedback information, the feedback information is generated based on the output travel inquiry information, and the travel information includes at least: travel destination and travel time; based on the meteorological information within the second time period, determining the target meteorological information corresponding to the travel destination at the travel time; and displaying the target meteorological information in the interactive interface.

[0101] In an optional embodiment, the user's travel plans can be proactively inquired about, and the user can provide feedback on the travel. After the user receives the feedback, the travel information can be determined based on the feedback. The feedback information can include the travel time and destination provided in voice or text. In another optional embodiment, the travel information can be determined based on a travel schedule prepared in advance by the user. The travel schedule can include the travel time and destination.

[0102] In another optional embodiment, after obtaining the travel information, the weather conditions corresponding to the travel destination at the travel time can be determined, and the weather conditions can be displayed in the interactive interface to remind the user to prepare the items they need to bring.

[0103] The following combination Figure 5 and Figure 6 A preferred embodiment of the present application is described in detail. The method can be executed by a computer terminal or a server. Figure 5 As shown, the method includes the following steps:

[0104] Step S501: Obtain a radar sequence, and use an encoder to extract features from the radar sequence to obtain a second feature;

[0105] The above-mentioned radar sequence may be a historical image sequence of the first time period.

[0106] Step S502: performing prediction processing on the second feature and decoding the prediction processing result using a decoder to obtain a future radar sequence;

[0107] The above-mentioned future radar sequence may be a historical image sequence of the second time period obtained by the first prediction.

[0108] In an optional embodiment, the u2net network may be used to perform prediction processing on the second feature.

[0109] Step S503, retrieving a first feature from a memory bank according to the future radar sequence;

[0110] The above-mentioned first feature may be a long-term feature, wherein the first feature may be obtained from a long-term memory bank. The long-term memory bank may be in a matrix form, where each row in the matrix is ​​a vector for representing a motion feature.

[0111] The above retrieval process is divided into two stages. Stage one can be used to update the memory bank, such as Figure 6The long sequence shown is composed of the input sequence and the output sequence. The long sequence can be output to the c3d encoder, and the features in the long sequence can be extracted using a 3D convolutional neural network. Then, features are extracted from each spatial position of the feature, and the long time series features are obtained by querying the memory bank, that is, the first feature mentioned above. After obtaining the first feature, the first feature can be fused with the second feature, and the fused feature can be predicted using an image prediction model.

[0112] The matrix in the above memory bank can be updated, and its purpose is to store long sequences of motion information in it.

[0113] The specific operation process of the above query is: set the vector corresponding to each pixel of the C3D coding feature to z i，j , memory bank is represented by Each row represents a vector, the number of vectors is L, and the vector dimension is c. i，j Calculate the similarity w with each row of M, as shown in the following formula (1), where d(.,.) represents the cosine function and the similarity between the two vectors. Then, the calculated similarity w is multiplied and added with m to obtain the long-term feature f, which is the first feature mentioned above.

[0114]

[0115] Among them, z i，j is the vector corresponding to the pixel in row i and column j, m l is the feature of row l, and d(.,.) is the cosine function.

[0116] The long time series feature f corresponding to the i-th row and j-th column i，j As shown in the following formula (2):

[0117]

[0118] Among them, l represents the number of vectors, w l is the similarity of l rows, m l is the feature of row l.

[0119] like Figure 6 As shown, in stage two, long-term time series features can also be retrieved. The input sequence (short sequence) is fed into another C3D encoder to extract features, retrieve the corresponding long-term time series features from memory, and fuse the extracted features with the long-term time series features for prediction. In this stage, the memory bank parameters are fixed and not updated; they are used only for feature retrieval.

[0120] It should be noted that, in each iteration, the above-mentioned phase 1 and phase 2 are performed alternately.

[0121] Step S504: Use the image prediction model to predict the second feature and the first feature to obtain a target radar sequence.

[0122] The above-mentioned target radar sequence is more accurate than the above-mentioned future radar sequence.

[0123] Example 2

[0124] According to an embodiment of the present application, an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than herein.

[0125] Figure 7 This is a flow chart of an image processing method according to embodiment 2 of the present application. Figure 7 As shown, the method may include the following steps:

[0126] Step S702: Acquire a historical weather map sequence.

[0127] The historical weather map sequence is a weather map sequence collected within the first time period.

[0128] Step S704 : using the meteorological prediction model to perform prediction processing on the historical meteorological image sequence based on the first feature corresponding to the historical meteorological image sequence, to obtain a target meteorological image sequence within the second time period.

[0129] The first feature is used to characterize the motion state of the meteorological targets included in the historical meteorological map sequence within a first time period and a second time period, and the second time period is a time period after the first time period.

[0130] The above-mentioned weather map sequence can be a weather map sequence of clouds, rain, snow, wind, etc.

[0131] In the above embodiment of the present application, the meteorological prediction model includes: a prediction network and a memory module, wherein the meteorological prediction model is used to perform prediction processing on the historical meteorological map sequence based on the first feature corresponding to the historical meteorological map sequence to obtain the target meteorological map sequence within the second time period, including: using the prediction network to extract features of the historical meteorological map sequence to obtain the second feature corresponding to the historical meteorological map sequence, wherein the second feature is used to characterize the motion state of the meteorological target within the first time period; obtaining the first feature stored in the memory module; and using the prediction network to perform prediction processing on the first feature and the second feature to obtain the target meteorological map sequence within the second time period.

[0132] In the above embodiment of the present application, after using the meteorological prediction model to predict the historical meteorological map sequence based on the first feature corresponding to the historical meteorological map sequence to obtain the target meteorological map sequence within the second time period, the method also includes: determining the meteorological information within the second time period based on the target meteorological map sequence; determining the reminder information corresponding to the meteorological information, wherein the reminder information can be used to prompt agricultural production information; and outputting the reminder information.

[0133] The above-mentioned meteorological information may include the amount of rainfall, the amount of snow, the intensity of the sun's ultraviolet rays, and the like.

[0134] The above reminder information can be presented in the form of an interface display, a text message prompt, or a voice playback.

[0135] The agricultural information may be fertilization information, pesticide spraying information, crop planting information, etc. In this application, the agricultural information is described by taking fertilization information as an example.

[0136] In an optional embodiment, whether there is a small amount of rainfall in the second time period can be determined based on the target meteorological image sequence. Since fertilizing crops is generally most effective on rainy days, if there is a small amount of rainfall in the second time period, the agricultural production information in the second time period is determined to be suitable for fertilization. At this time, the agricultural production information suitable for fertilization can be displayed in the interactive interface to prompt the user to fertilize; further, the information suitable for fertilization can be broadcast in the form of voice broadcast; further, the information suitable for fertilization can be sent to the user's mobile terminal via text message.

[0137] In the above embodiment of the present application, the image prediction model also includes: a feature extraction network, and obtaining the first feature stored in the memory module includes: using the feature extraction network to process the historical image sequence to obtain a third feature corresponding to the historical image sequence; obtaining the first vector corresponding to each spatial position in the third feature; and obtaining the first feature based on the first vector corresponding to each spatial position and multiple second vectors stored in the memory module.

[0138] In the above embodiment of the present application, the prediction network includes: an encoder composed of multiple encoding layers, and a decoder composed of multiple decoding layers, the output of the memory module is connected to the first decoding layer of the decoder, and the multiple encoding layers are connected to the multiple decoding layers in a one-to-one correspondence, wherein the encoder is used to extract features from the historical image sequence to obtain the second feature, and the decoder is used to perform prediction processing on the first feature and the second feature to obtain the target weather map sequence.

[0139] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0140] Example 3

[0141] According to an embodiment of the present application, an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than herein.

[0142] Figure 8 This is a flow chart of an image processing method according to embodiment 3 of the present application. Figure 8 As shown, the method may include the following steps:

[0143] Step S802: The cloud server obtains a historical image sequence.

[0144] The historical image sequence is an image sequence collected within the first time period.

[0145] In step S804 , the cloud server uses the image prediction model to perform prediction processing on the historical image sequence based on the first feature corresponding to the historical image sequence to obtain a target image sequence within the second time period.

[0146] The first feature is used to characterize the motion state of the target contained in the historical image sequence within a first time period and a second time period, and the second time period is a time period after the first time period.

[0147] In the above embodiment of the present application, the image prediction model includes: a prediction network and a memory module, wherein the image prediction model is used to perform prediction processing on the historical image sequence based on the first feature corresponding to the historical image sequence to obtain the target image sequence within the second time period, including: the cloud server uses the prediction network to extract features from the historical image sequence to obtain the second feature corresponding to the historical image sequence, wherein the second feature is used to characterize the motion state of the target within the first time period; the cloud server obtains the first feature stored in the memory module; the cloud server uses the prediction network to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0148] In the above embodiment of the present application, the image prediction model also includes: a feature extraction network, and obtaining the first feature stored in the memory module includes: the cloud server uses the feature extraction network to process the historical image sequence to obtain a third feature corresponding to the historical image sequence; the cloud server obtains the first vector corresponding to each spatial position in the third feature; the cloud server obtains the first feature based on the first vector corresponding to each spatial position and the multiple second vectors stored in the memory module.

[0149] In the above embodiment of the present application, the prediction network includes: an encoder composed of multiple encoding layers, and a decoder composed of multiple decoding layers, the output of the memory module is connected to the first decoding layer of the decoder, and the multiple encoding layers are connected to the multiple decoding layers in a one-to-one correspondence, wherein the encoder is used to extract features from the historical image sequence to obtain the second feature, and the decoder is used to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0150] In the above embodiment of the present application, the target image sequence is a weather map sequence. After using the image prediction model to predict the historical image sequence based on the first feature corresponding to the historical image sequence to obtain the target image sequence within the second time period, the method also includes: the cloud server determines the weather information within the second time period based on the target image sequence; the cloud server displays the weather information in the interactive interface.

[0151] In the above embodiment of the present application, after determining the meteorological information within the second time period based on the target image sequence, the method also includes: the cloud server generates travel recommendation information within the second time period based on the meteorological information, wherein the travel recommendation information includes at least one of the following: travel route, travel time and items to be carried; the cloud server displays the travel recommendation information in the interactive interface.

[0152] In the above embodiment of the present application, after determining the meteorological information within the second time period based on the target image sequence, the method also includes: the cloud server obtains travel information, wherein the travel information is determined based on the travel schedule or feedback information, the feedback information is generated based on the output travel inquiry information, and the travel information includes at least: travel destination and travel time; the cloud server determines the target meteorological information corresponding to the travel destination at the travel time based on the meteorological information within the second time period; the cloud server displays the target meteorological information in the interactive interface.

[0153] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0154] Example 4

[0155] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Figure 9 As shown, the device 900 includes: a first acquisition module 902 and a first prediction module 904.

[0156] Among them, the first acquisition module is used to obtain a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; the first prediction module is used to use an image prediction model to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and the second time period, and the second time period is the time period after the first time period.

[0157] It should be noted that the first acquisition module 902 and the first prediction module 904 correspond to steps S202 to S204 of Example 1. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0158] In the above embodiment of the present application, the image prediction model includes: a prediction network and a memory module, wherein the image prediction model is based on the first feature corresponding to the historical image sequence, and the first prediction module includes: a first extraction unit, a first acquisition unit, and a first prediction unit.

[0159] Among them, the first extraction unit is used to use the prediction network to extract features of the historical image sequence to obtain a second feature corresponding to the historical image sequence, wherein the second feature is used to characterize the motion state of the target within the first time period; the first acquisition unit is used to obtain the first feature stored in the memory module; the first prediction unit is used to use the prediction network to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0160] In the above embodiment of the present application, the image prediction model also includes: a feature extraction network, a first acquisition unit, including: a first processing subunit, a first acquisition subunit, and a storage unit.

[0161] Among them, the first processing subunit is used to process the historical image sequence using the feature extraction network to obtain the third feature corresponding to the historical image sequence; the first acquisition subunit is used to obtain the first vector corresponding to each spatial position in the third feature; the storage unit is used to obtain the first feature based on the first vector corresponding to each spatial position and the multiple second vectors stored in the memory module.

[0162] In the above embodiment of the present application, the prediction network includes: an encoder composed of multiple encoding layers, and a decoder composed of multiple decoding layers, the output of the memory module is connected to the first decoding layer of the decoder, and the multiple encoding layers are connected to the multiple decoding layers in a one-to-one correspondence, wherein the encoder is used to extract features from the historical image sequence to obtain the second feature, and the decoder is used to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0163] In the above embodiment of the present application, the device further includes: a first determination module and a first display module.

[0164] The first determination module is used to determine the meteorological information within the second time period based on the target image sequence; and the first display module is used to display the meteorological information in the interactive interface.

[0165] In the above embodiments of the present application, the device further includes: a generating module.

[0166] Among them, the generation module is used to generate travel recommendation information within the second time period based on meteorological information, wherein the travel recommendation information includes at least one of the following: travel route, travel time and items to be carried; the first display module is also used to display the travel recommendation information in the interactive interface.

[0167] Among them, the first acquisition module is also used to obtain travel information, wherein the travel information is determined based on the travel schedule or feedback information, the feedback information is generated based on the output travel inquiry information, and the travel information includes at least: travel destination and travel time; the first determination module is also used to determine the target meteorological information corresponding to the travel destination at the travel time based on the meteorological information in the second time period; the first display module is also used to display the target meteorological information in the interactive interface.

[0168] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0169] Example 5

[0170] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Figure 10 As shown, the device 1000 includes: a second acquisition module 1002 and a second prediction module 1004.

[0171] Among them, the second acquisition module is used to obtain a historical meteorological map sequence, wherein the historical meteorological map sequence is a meteorological map sequence collected within a first time period; the second prediction module is used to use a meteorological prediction model to perform prediction processing on the historical meteorological map sequence based on a first feature corresponding to the historical meteorological map sequence, to obtain a target meteorological map sequence within a second time period, wherein the first feature is used to characterize the movement state of the meteorological target contained in the historical meteorological map sequence within the first time period and the second time period, and the second time period is the time period after the first time period.

[0172] It should be noted that the second acquisition module 1002 and the second prediction module 1004 correspond to steps S702 to S704 of Example 2. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0173] In the above embodiment of the present application, the weather forecast model includes: a forecast network and a memory module, wherein the weather forecast model utilizes the first feature corresponding to the historical weather map sequence, and the second forecast module includes: a second extraction unit, a second acquisition unit, and a second forecast unit.

[0174] Among them, the second extraction unit is used to use the prediction network to extract features of the historical meteorological image sequence to obtain the second feature corresponding to the historical meteorological image sequence, wherein the second feature is used to characterize the motion state of the meteorological target within the first time period; the second acquisition unit is used to obtain the first feature stored in the memory module; the second prediction unit is used to use the prediction network to perform prediction processing on the first feature and the second feature to obtain the target image sequence within the second time period.

[0175] In the above embodiment of the present application, the device further includes: a second determination module and an output module.

[0176] Among them, the second determination module is used to determine the meteorological information within the second time period based on the target meteorological map sequence; the second determination module is also used to determine the reminder information corresponding to the meteorological information, wherein the reminder information can be used to prompt agricultural production information; the output module is used to output the reminder information.

[0177] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0178] Example 6

[0179] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Figure 11As shown, the device 1100 includes: a third acquisition module 1102 and a third prediction module 1104.

[0180] Among them, the third acquisition module is used to obtain a historical image sequence through a cloud server, wherein the historical image sequence is an image sequence collected within a first time period; the third prediction module is used to use an image prediction model through a cloud server to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, and the second time period is the time period after the first time period.

[0181] It should be noted that the third acquisition module 1102 and the third prediction module 1104 described above correspond to steps S802 to S804 of Example 3. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0182] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0183] Example 7

[0184] The embodiment of the present application can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0185] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0186] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image feature extraction method: obtaining a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; performing feature extraction on the historical image sequence to obtain a second feature corresponding to the historical image sequence, wherein the second feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period; obtaining a first feature corresponding to the historical image sequence, wherein the first feature is used to characterize the motion state of the target within the first time period and a second time period, and the second time period is a time period after the first time period; performing prediction processing on the second feature and the first feature to obtain a target image sequence within the second time period.

[0187] Optionally, Figure 12 FIG. 1 is a structural block diagram of a computer terminal according to an embodiment of the present application. Figure 12 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1202 and a memory 1204.

[0188] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image feature extraction method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned image feature extraction method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0189] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; using an image prediction model to predict the historical image sequence based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and the second time period, and the second time period is the time period after the first time period.

[0190] Optionally, the processor may also execute the program code of the following steps: performing feature extraction on the historical image sequence using a prediction network to obtain a second feature corresponding to the historical image sequence, wherein the second feature is used to characterize the motion state of the target within a first time period; obtaining the first feature stored in the memory module; and performing prediction processing on the first feature and the second feature using a prediction network to obtain a target image sequence.

[0191] Optionally, the processor may also execute the program code of the following steps: using a feature extraction network to process the historical image sequence to obtain a third feature corresponding to the historical image sequence; obtaining a first vector corresponding to each spatial position in the third feature; and obtaining a first feature based on the first vector corresponding to each spatial position and multiple second vectors stored in the memory module.

[0192] Optionally, the processor may also execute the program code of the following steps: the prediction network includes: an encoder composed of multiple encoding layers, and a decoder composed of multiple decoding layers, the output of the memory module is connected to the first decoding layer of the decoder, and the multiple encoding layers are connected to the multiple decoding layers in a one-to-one correspondence, wherein the encoder is used to extract features from the historical image sequence to obtain the second feature, and the decoder is used to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0193] Optionally, the processor may further execute program codes of the following steps: determining meteorological information within a second time period based on the target image sequence; and displaying the meteorological information in an interactive interface.

[0194] Optionally, the processor may also execute the program code of the following steps: generating travel recommendation information within the second time period based on meteorological information, wherein the travel recommendation information includes at least one of the following: travel route, travel time, and items to be carried; and displaying the travel recommendation information in the interactive interface.

[0195] Optionally, the processor may also execute the program code of the following steps: obtaining travel information, wherein the travel information is determined based on the travel schedule or feedback information, the feedback information is generated based on the output travel inquiry information, and the travel information includes at least: the travel destination and the travel time; based on the weather information in the second time period, determining the target weather information corresponding to the travel destination at the travel time; and displaying the target weather information in the interactive interface.

[0196] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining a historical meteorological map sequence, wherein the historical meteorological map sequence is a meteorological map sequence collected within a first time period; using a meteorological prediction model to predict the historical meteorological map sequence based on a first feature corresponding to the historical meteorological map sequence, to obtain a target meteorological map sequence within a second time period, wherein the first feature is used to characterize the motion state of the meteorological target contained in the historical meteorological map sequence within the first time period and the second time period, and the second time period is the time period after the first time period.

[0197] Optionally, the processor may also execute the program code of the following steps: performing feature extraction on a historical meteorological image sequence using a prediction network to obtain a second feature corresponding to the historical meteorological image sequence, wherein the second feature is used to characterize the motion state of the meteorological target within a first time period; obtaining the first feature stored in the memory module; and performing prediction processing on the first feature and the second feature using a prediction network to obtain a target image sequence within a second time period.

[0198] Optionally, the processor may also execute the program code of the following steps: determining the meteorological information within the second time period based on the target meteorological map sequence; determining the reminder information corresponding to the meteorological information, wherein the reminder information may be used to prompt agricultural production information; and outputting the reminder information.

[0199] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: the cloud server obtains a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; the cloud server uses an image prediction model to predict the historical image sequence based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and the second time period, and the second time period is the time period after the first time period.

[0200] It can be understood by those skilled in the art that Figure 12 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 12 It does not limit the structure of the above electronic device. For example, the computer terminal A may also include Figure 12 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 12 Different configurations shown.

[0201] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0202] Example 12

[0203] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image segmentation method provided in the above embodiment.

[0204] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0205] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; using an image prediction model to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, and the second time period is the time period after the first time period.

[0206] Optionally, the storage medium is further configured to store program codes for executing the following steps: extracting features from a historical image sequence using a prediction network to obtain a second feature corresponding to the historical image sequence, wherein the second feature is used to characterize the motion state of the target within a first time period; obtaining the first feature stored in the memory module; and performing prediction processing on the first feature and the second feature using a prediction network to obtain a target image sequence.

[0207] Optionally, the storage medium is also configured to store program codes for executing the following steps: processing the historical image sequence using a feature extraction network to obtain a third feature corresponding to the historical image sequence; obtaining a first vector corresponding to each spatial position in the third feature; and obtaining the first feature based on the first vector corresponding to each spatial position and multiple second vectors stored in the memory module.

[0208] Optionally, the storage medium is further configured to store program code for executing the following steps: the prediction network includes: an encoder composed of multiple encoding layers, and a decoder composed of multiple decoding layers, the output of the memory module is connected to the first decoding layer of the decoder, and the multiple encoding layers are connected to the multiple decoding layers in a one-to-one correspondence, wherein the encoder is used to extract features from the historical image sequence to obtain the second feature, and the decoder is used to perform prediction processing on the first feature and the second feature to obtain the target image sequence.

[0209] Optionally, the storage medium is further configured to store program codes for executing the following steps: determining meteorological information within a second time period based on the target image sequence; and displaying the meteorological information in an interactive interface.

[0210] Optionally, the above-mentioned storage medium is also configured to store program code for executing the following steps: generating travel recommendation information within a second time period based on meteorological information, wherein the travel recommendation information includes at least one of the following: travel route, travel time and items to be carried; and displaying the travel recommendation information in an interactive interface.

[0211] Optionally, the above-mentioned storage medium is also configured to store program code for executing the following steps: obtaining travel information, wherein the travel information is determined based on the travel schedule or feedback information, the feedback information is generated based on the output travel inquiry information, and the travel information includes at least: the travel destination and travel time; based on the meteorological information in the second time period, determining the target meteorological information corresponding to the travel destination at the travel time; and displaying the target meteorological information in the interactive interface.

[0212] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a historical meteorological map sequence, wherein the historical meteorological map sequence is a meteorological map sequence collected within a first time period; using a meteorological prediction model to perform prediction processing on the historical meteorological map sequence based on a first feature corresponding to the historical meteorological map sequence, to obtain a target meteorological map sequence within a second time period, wherein the first feature is used to characterize the motion state of the meteorological target contained in the historical meteorological map sequence within the first time period and a second time period, and the second time period is the time period after the first time period.

[0213] Optionally, the above-mentioned storage medium is also configured to store program codes for executing the following steps: using a prediction network to extract features of a historical meteorological map sequence to obtain a second feature corresponding to the historical meteorological map sequence, wherein the second feature is used to characterize the motion state of the meteorological target within a first time period; obtaining the first feature stored in the memory module; using a prediction network to perform prediction processing on the first feature and the second feature to obtain a target image sequence within a second time period.

[0214] Optionally, the above-mentioned storage medium is also configured to store program code for executing the following steps: determining the meteorological information within the second time period based on the target meteorological map sequence; determining the reminder information corresponding to the meteorological information, wherein the reminder information can be used to prompt agricultural production information; and outputting the reminder information.

[0215] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: the cloud server obtains a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; the cloud server uses an image prediction model to perform prediction processing on the historical image sequence based on a first feature corresponding to the historical image sequence, to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and a second time period, and the second time period is the time period after the first time period.

[0216] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of images involved in the above embodiments of this application are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0217] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0219] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0220] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0221] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0222] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; Extracting features from the historical image sequence using a prediction network to obtain a second feature corresponding to the historical image sequence; Processing the historical image sequence using a feature extraction network to obtain a third feature corresponding to the historical image sequence; Obtaining a first feature based on a first vector corresponding to each spatial position in the third feature and a plurality of second vectors stored in a memory module; The prediction network is used to perform prediction processing on the first feature and the second feature to obtain a target image sequence, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and the second time period, and the second time period is the time period after the first time period.

2. The method according to claim 1, characterized in that The second feature is used to characterize the motion state of the target in the first time period.

3. The method according to claim 2, characterized in that The prediction network includes: an encoder composed of multiple encoding layers, and a decoder composed of multiple decoding layers, the output of the memory module is connected to the first decoding layer of the decoder, and the multiple encoding layers are connected to the multiple decoding layers in a one-to-one correspondence, wherein the encoder is used to extract features from the historical image sequence to obtain the second features, and the decoder is used to perform prediction processing on the first features and the second features to obtain the target image sequence.

4. The method according to any one of claims 1 to 3, characterized in that The target image sequence is a weather image sequence. After performing prediction processing on the historical image sequence using an image prediction model based on a first feature corresponding to the historical image sequence to obtain a target image sequence within a second time period, the method further includes: determining meteorological information within the second time period based on the target image sequence; The weather information is displayed in an interactive interface.

5. The method according to claim 4, characterized in that After determining the meteorological information within the second time period based on the target image sequence, the method further includes: generating travel recommendation information for the second time period based on the weather information, wherein the travel recommendation information includes at least one of the following: travel route, travel time, and items to be carried; The travel recommendation information is displayed in the interactive interface.

6. The method according to claim 4, characterized in that After determining the meteorological information within the second time period based on the target image sequence, the method further includes: Acquiring travel information, wherein the travel information is determined based on a travel schedule or feedback information, the feedback information is generated based on the output travel inquiry information, and the travel information includes at least: a travel destination and a travel time; Determining target weather information corresponding to the travel destination at the travel time based on the weather information during the second time period; The target meteorological information is displayed in the interactive interface.

7. An image processing method, characterized in that: include: Acquire a historical weather map sequence, wherein the historical weather map sequence is a weather map sequence collected within a first time period; Extracting features from the historical meteorological image sequence using a prediction network to obtain a second feature corresponding to the historical meteorological image sequence, wherein the second feature is used to characterize a motion state of a meteorological target within the first time period; Obtaining a first feature stored in a memory module; Processing the historical meteorological image sequence using a feature extraction network to obtain a third feature corresponding to the historical meteorological image sequence; Obtaining the first feature based on a first vector corresponding to each spatial position in the third feature and a plurality of second vectors stored in the memory module; The prediction network is used to perform prediction processing on the first feature and the second feature to obtain a target image sequence within a second time period, wherein the first feature is used to characterize the motion state of the meteorological targets contained in the historical meteorological image sequence within the first time period and the second time period, and the second time period is the time period after the first time period.

8. The method according to claim 7, characterized in that The second feature is used to characterize the motion state of the meteorological target in the first time period.

9. The method according to claim 7, characterized in that After performing prediction processing on the historical meteorological image sequence using a meteorological prediction model based on the first feature corresponding to the historical meteorological image sequence to obtain a target meteorological image sequence within a second time period, the method further includes: determining meteorological information within the second time period based on the target meteorological map sequence; Determining that the meteorological information corresponds to reminder information, wherein the reminder information is used to prompt agricultural production information; The reminder information is output.

10. An image processing method, characterized in that: include: The cloud server obtains a historical image sequence, wherein the historical image sequence is an image sequence collected within a first time period; The server uses a prediction network to extract features from the historical image sequence to obtain a second feature corresponding to the historical image sequence; The server processes the historical image sequence using a feature extraction network to obtain a third feature corresponding to the historical image sequence; The server obtains the first feature based on the first vector corresponding to each spatial position in the third feature and the plurality of second vectors stored in the memory module; The server uses the prediction network to perform prediction processing on the first feature and the second feature to obtain a target image sequence, wherein the first feature is used to characterize the motion state of the target contained in the historical image sequence within the first time period and the second time period, and the second time period is a time period after the first time period.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 10.

12. A computer terminal, characterized in that: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a program that can be executed by the at least one processor, and when the at least one processor executes the program, the image processing method according to any one of claims 1 to 10 is executed.

Images