Configuration method and device of optical flow estimation model, equipment and storage medium
By generating multimodal image data and pre-training, fine-tuning, and feature fusion training of the optical flow estimation model, the problem of insufficient generalization ability of optical flow estimation methods in multimodal image data is solved, and more efficient cross-modal optical flow estimation is achieved.
Patent Information
- Application Number
- CN202511023797.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-23
- Publication Date
- 2025-12-12
AI Technical Summary
Existing optical flow estimation methods have poor generalization ability in multimodal image data and are difficult to effectively handle modal differences between different imaging systems or styles.
Multimodal image data is generated by a multimodal optical flow data generation module. The optical flow estimation model is pre-trained, fine-tuned, and trained using feature fusion based on the multimodal image data. The model is then optimized by combining feature distillation technology to enhance its cross-modal matching capability.
This improves the optical flow estimation model's ability to perceive images of different modalities and its target tracking accuracy, thereby enhancing the system's environmental perception capabilities.
Smart Images
Figure CN121120706A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and more specifically, to a configuration method, apparatus, device, and storage medium for an optical flow estimation model. Background Technology
[0002] Optical flow estimation refers to estimating the motion vector of a pixel in a continuous image sequence by analyzing the pixel's motion. In multimodal image data perception scenarios, especially in dynamic environments, optical flow estimation plays a crucial role.
[0003] However, optical flow estimation faces significant challenges due to modal differences between different imaging systems or styles. For example, images from different modalities, such as RGB images, infrared images, depth images, and event camera images, exhibit significant differences in pixel intensity distribution, imaging principles, and noise characteristics. Existing optical flow estimation methods typically focus on extracting modal-invariant features and are trained on limited datasets, resulting in poor generalization ability. Summary of the Invention
[0004] According to one aspect of this application, a method for configuring an optical flow estimation model is provided, comprising: generating multimodal image data based on a preset multimodal optical flow data generation module; training a preset optical flow estimation model using the multimodal image data; and optimizing the trained optical flow estimation model.
[0005] According to some embodiments, multimodal image data is generated based on a preset multimodal optical flow data generation module, including: acquiring image source data through the multimodal optical flow data generation module; acquiring image guidance data through the multimodal optical flow data generation module; and generating multimodal image data based on the image source data and the image guidance data through the multimodal optical flow data generation module.
[0006] According to some embodiments, obtaining image source data through a multimodal optical flow data generation module includes: obtaining a preset multi-view image dataset; and preprocessing the multi-view image dataset through the multimodal optical flow data generation module to obtain image source data.
[0007] According to some embodiments, image guidance data is obtained through a multimodal optical flow data generation module, including: obtaining a preset cross-modal image dataset; and preprocessing the cross-modal image dataset through the multimodal optical flow data generation module to obtain image guidance data.
[0008] According to some embodiments, multimodal image data is generated by a multimodal optical flow data generation module based on image source data and image guidance data, including: configuring a multimodal generation model through the multimodal optical flow data generation module; and processing the image source data through the multimodal generation model based on the image guidance data to generate multimodal image data corresponding to the multimodal generation model.
[0009] According to some embodiments, training a preset optical flow estimation model using multimodal image data includes: pre-training the optical flow estimation model to obtain initial parameters; and fine-tuning and feature fusion training the pre-trained optical flow estimation model using multimodal image data based on the initial parameters.
[0010] According to some embodiments, the optical flow estimation model is pre-trained to obtain initial parameters, including: acquiring image source data; pre-training the optical flow estimation model using the image source data; and optimizing the parameters of the pre-trained optical flow estimation model based on a preset optical flow estimation loss function to obtain initial parameters.
[0011] According to some embodiments, based on initial parameters, a pre-trained optical flow estimation model is fine-tuned and feature fusion trained using multimodal image data, including: incremental processing of the multimodal image data; fine-tuning the pre-trained optical flow estimation model using the incrementally processed multimodal image data at a preset learning rate based on the initial parameters to update the parameters of the optical flow estimation model; and adjusting the parameter update frequency and learning rate of the optical flow estimation model.
[0012] According to some embodiments, based on initial parameters, a pre-trained optical flow estimation model is fine-tuned and its features are fused using multimodal image data. This includes: incrementally processing the multimodal image data; inputting the incrementally processed multimodal image data into a preset neural network in the optical flow estimation model to obtain multimodal optical flow feature prediction results; iteratively training the preset neural network based on the multimodal optical flow feature prediction results to obtain fused multimodal optical flow features; performing feature alignment on the fused multimodal optical flow features; calculating the feature consistency loss of the fused multimodal optical flow features based on a preset feature consistency loss function to update the parameters of the optical flow estimation model; and adjusting the parameter update frequency of the optical flow estimation model.
[0013] According to some embodiments, optimizing a trained optical flow estimation model includes: performing feature distillation on the trained optical flow estimation model to obtain intermediate layer features of the optical flow estimation model; and transferring the intermediate layer features of the optical flow estimation model to a preset lightweight model to optimize the optical flow estimation model.
[0014] According to one aspect of this application, a configuration device for an optical flow estimation model is provided, comprising: a data generation unit for generating multimodal image data based on a preset multimodal optical flow data generation module; a model training unit for training a preset optical flow estimation model using the multimodal image data; and a model optimization unit for optimizing the trained optical flow estimation model.
[0015] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method as described above.
[0016] According to one aspect of this application, a computer-readable storage medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.
[0017] According to embodiments of this application, high-quality multimodal image data can be provided for training optical flow estimation models, enhancing the optical flow estimation models' ability to perceive different modalities of images, which is of great significance for improving the system's environmental perception capability and target tracking accuracy.
[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application.
[0020] Figure 1 A flowchart illustrating a configuration method for an optical flow estimation model according to an example embodiment of this application is shown.
[0021] Figure 2 A schematic diagram of a configuration apparatus for an optical flow estimation model according to an example embodiment of this application is shown.
[0022] Figure 3 A block diagram of an electronic device according to an example embodiment of this application is shown. Detailed Implementation
[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.
[0024] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.
[0025] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0026] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0027] This application provides a configuration method, apparatus, device, and storage medium for an optical flow estimation model, which can provide high-quality multimodal image data and enhance the optical flow estimation model's ability to perceive images of different modalities.
[0028] The following will describe in detail, with reference to the accompanying drawings, a method, apparatus, device, and storage medium for configuring an optical flow estimation model according to an embodiment of this application.
[0029] Figure 1 A flowchart illustrating a configuration method for an optical flow estimation model according to an example embodiment of this application is shown.
[0030] like Figure 1 As shown, in step S100, multimodal image data is generated based on a preset multimodal optical flow data generation module.
[0031] For example, in step S100, the configuration device acquires a preset multimodal optical flow data generation module and generates multimodal image data through the multimodal optical flow data generation module.
[0032] The configuration device acquires a preset multimodal optical flow data generation module.
[0033] According to some embodiments, the multimodal optical flow data generation module includes a source data submodule, a guide data submodule, and a generation model submodule.
[0034] The configuration device acquires a preset multi-view image dataset.
[0035] According to some embodiments, the multi-view image dataset can employ the large-scale RGB image dataset Tartanair to ensure data diversity and richness.
[0036] The configuration device preprocesses the multi-view image dataset through the source data submodule of the multimodal optical flow data generation module to obtain image source data.
[0037] According to some embodiments, the configuration device can crop, scale, and normalize RGB images in a multi-view image dataset through a source data submodule to adapt to the input requirements of a multimodal generation model and remove noise and outliers from the images to improve data quality.
[0038] The configuration device determines the target modal image and acquires a preset cross-modal image dataset.
[0039] According to some embodiments, the cross-modal image dataset may employ publicly available cross-modal datasets, such as LLVIP and M3FD, to ensure that the multimodal generative model can learn the mapping relationship between different modal data.
[0040] The configuration device preprocesses the cross-modal image dataset through the guidance data submodule of the multimodal optical flow data generation module to obtain image guidance data.
[0041] According to some embodiments, the configuration device can perform alignment processing on RGB images in a cross-modal image dataset through a guide data submodule to ensure spatial consistency between the RGB images and the target modal image. Furthermore, the configuration device can perform data augmentation (such as rotation, flipping, brightness adjustment, etc.) on the RGB images in the cross-modal image dataset through the guide data submodule to increase data diversity.
[0042] The configuration device configures the multimodal generation model through the generation model submodule of the multimodal optical flow data generation module.
[0043] According to some embodiments, the configuration device acquires multiple preset generative models and configures the multiple generative models into multimodal generative models of different modalities through a generative model submodule, including an infrared modal generative model, a depth modal generative model, an event modal generative model, a normal modal generative model, and an art style modal generative model.
[0044] According to some embodiments, the infrared modality generation model can employ a diffusion-based StyleBooth model to fine-tune the infrared style of RGB images, converting RGB images into thermal imaging-style infrared images while maintaining structural consistency.
[0045] According to some embodiments, the depth modality generation model may employ the DepthAnything V2 model to predict accurate depth information from RGB images.
[0046] According to some embodiments, the event modality generation model can simulate the grayscale change rate based on the event camera imaging model to generate event map sequences similar to those of a real event camera.
[0047] According to some embodiments, the normal mode generation model obtains the depth information of the RGB image through the depth mode generation model, calculates the surface normal direction of each pixel in the image, and generates a normal map.
[0048] According to some embodiments, the art style modality generation model can employ a style transfer network model, such as AdaIN or STROTSS, to perform specific art style transfer on RGB images, simulate artistic modal input, and generate images with specific art styles, such as oil painting style or sketch style.
[0049] Based on image-guided data, the configuration device processes the image source data through a multimodal generation model to generate multimodal image data corresponding to the multimodal generation model.
[0050] In step S200, a preset optical flow estimation model is trained using multimodal image data.
[0051] For example, in step S200, the configuration device trains the optical flow estimation model using multimodal image data generated by the multimodal optical flow data generation module.
[0052] The configuration device acquires image source data through the multimodal optical flow data generation module and uses it to pre-train a preset optical flow estimation model.
[0053] According to some embodiments, the preset optical flow estimation model can employ matching models such as LightGlue, LoFTR, or RoMa, enabling cross-modal feature extraction. The configuration device pre-trains the optical flow estimation model using image source data, so that the optical flow estimation model can accurately estimate the optical flow between RGB images.
[0054] Based on a preset optical flow estimation loss function, the configuration device optimizes the parameters of the pre-trained optical flow estimation model to obtain initial parameters.
[0055] According to some embodiments, the optical flow estimation loss function uses Euclidean distance (L2 norm) or Manhattan distance (L1 norm) to calculate the error between the predicted optical flow and the actual optical flow. The optical flow estimation loss function L... flow It can be expressed by the following formula.
[0056]
[0057] Where H represents the image height, W represents the image width, i represents the pixel index in the image, and f gt,i Represents the actual optical flow, f pred,i This represents the predicted optical flow.
[0058] According to some embodiments, the configuration device optimizes the parameters of the pre-trained optical flow estimation model using an optical flow estimation loss function to ensure good performance of the optical flow estimation model on RGB image data. Then, the configuration device obtains the optimized initial parameters of the optical flow estimation model.
[0059] After the optical flow estimation model is pre-trained, the configuration device performs incremental processing on the multimodal image data generated by the multimodal optical flow data generation module.
[0060] Furthermore, based on the initial parameters of the optical flow estimation model, the configuration device fine-tunes the optical flow estimation model using multimodal image data that has undergone incremental processing at a preset learning rate, so as to update the parameters of the optical flow estimation model.
[0061] According to some embodiments, the multimodal image data input to the optical flow estimation model in real time includes image data from different modalities. The fine-tuned optical flow estimation model has cross-modal matching capability and can output pixel-level dense optical flow maps or sparse matching points.
[0062] The configuration device adjusts the parameter update frequency and learning rate of the optical flow estimation model.
[0063] According to some embodiments, by performing real-time statistics on the input and output data of the optical flow estimation model, the configuration device can dynamically adjust the parameter update frequency and learning rate of the optical flow estimation model so that the optical flow estimation model can quickly adapt to environmental changes.
[0064] During the fine-tuning of the optical flow estimation model, the configuration device also inputs the incrementally processed multimodal image data into the preset neural network in the optical flow estimation model to obtain the multimodal optical flow feature prediction results.
[0065] According to some embodiments, the preset neural network in the optical flow estimation model can adopt an attention mechanism or a Transformer architecture, which can automatically learn the relationship between features of different modalities and fuse them into a unified feature representation.
[0066] According to some embodiments, the configuration device inputs two consecutive frames of multimodal image data into a preset neural network and obtains the displacement of each pixel in the image output by the preset neural network, i.e., the multimodal optical flow feature prediction result.
[0067] The configuration device iteratively trains a preset neural network based on the prediction results of multimodal optical flow features to obtain fused multimodal optical flow features.
[0068] According to some embodiments, the configuration device compares the predicted multimodal optical flow features with the actual optical flow features and calculates the error between them (e.g., endpoint error). Then, based on the calculated error, the configuration device adjusts the parameters of a preset neural network through backpropagation and iteratively trains the preset neural network using a large number of multimodal image data samples, so that the preset neural network can automatically estimate the optical flow in the image and obtain fused multimodal optical flow features.
[0069] The configuration device performs feature alignment on the fused multimodal optical flow features to ensure the spatial and semantic consistency of optical flow features of different modes.
[0070] According to some embodiments, the configuration device can perform feature alignment of fused multimodal optical flow features through a spatial transformation network (STN).
[0071] Based on a preset feature consistency loss function, the configuration device calculates the feature consistency loss of the fused multimodal optical flow features that have undergone feature alignment, in order to update the parameters of the optical flow estimation model.
[0072] According to some embodiments, the feature consistency loss function calculates the feature consistency loss using a similarity metric (e.g., cosine similarity or Euclidean distance) between features of image data from different modalities. Feature consistency loss function L consistency It can be expressed by the following formula.
[0073]
[0074] Where H represents the image height, W represents the image width, i represents the pixel index in the image, and F... ir,i F represents the features of image data of different modalities. rgb,i Characteristics representing RGB image data.
[0075] According to some embodiments, the configuration device adjusts the parameter update frequency of the optical flow estimation model by using the feature consistency loss of the fused multimodal optical flow features to complete the feature fusion training of the optical flow estimation model.
[0076] According to some embodiments, the configuration device is based on the optical flow estimation loss function L flow Feature-consistent loss function L consistencyThe total loss function L can be calculated using the following formula.
[0077] L=λL consistency +L flow (3)
[0078] Here, λ is an adaptive weighting coefficient used to balance the optical flow estimation loss and the feature consistency loss. By dynamically adjusting the value of λ, the optical flow estimation model can optimize different objectives during training.
[0079] In step S300, the trained optical flow estimation model is optimized.
[0080] For example, in step S300, the configuration device performs feature distillation on the trained optical flow estimation model to optimize the optical flow estimation model.
[0081] The configuration device identifies the trained optical flow estimation model as the teacher model and performs feature distillation on it to obtain the intermediate layer features of the optical flow estimation model.
[0082] The configuration device acquires a preset lightweight model to serve as the student model.
[0083] According to some embodiments, the lightweight model can adopt the Mamba model, which can significantly reduce the amount of computation while maintaining high accuracy.
[0084] The configuration device transfers intermediate layer features from the optical flow estimation model to a lightweight model to optimize the optical flow estimation model.
[0085] According to some embodiments, the configuration device calculates the similarity loss between the intermediate layer features of the trained optical flow estimation model and the lightweight model, and adds it to the total loss function L to optimize the optical flow estimation model. The similarity loss L distill It can be expressed by the following formula.
[0086] L distill =KL(F teacher ||F student (4)
[0087] Among them, F teacher F represents the intermediate layer features of the optical flow estimation model. student These are intermediate layer features for lightweight models.
[0088] According to embodiments of this application, high-quality multimodal image data can be provided for training an optical flow estimation model, thereby enhancing the optical flow estimation model's ability to perceive images of different modalities.
[0089] Figure 2A schematic diagram of a configuration apparatus for an optical flow estimation model according to an example embodiment of this application is shown.
[0090] like Figure 2 As shown, the configuration device 100 includes a data generation unit 110, a model training unit 120, and a model optimization unit 130.
[0091] The data generation unit 110 acquires the preset multimodal optical flow data generation module.
[0092] The data generation unit 110 acquires a preset multi-view image dataset.
[0093] The data generation unit 110 preprocesses the multi-view image dataset through the source data submodule of the multimodal optical flow data generation module to obtain image source data.
[0094] The data generation unit 110 determines the target modal image and acquires a preset cross-modal image dataset.
[0095] The data generation unit 110 preprocesses the cross-modal image dataset through the guide data submodule of the multimodal optical flow data generation module to obtain image guide data.
[0096] The data generation unit 110 configures the multimodal generation model through the generation model submodule of the multimodal optical flow data generation module.
[0097] Based on the image-guided data, the data generation unit 110 processes the image source data through a multimodal generation model to generate multimodal image data corresponding to the multimodal generation model.
[0098] The model training unit 120 acquires image source data through the multimodal optical flow data generation module and uses it to pre-train the preset optical flow estimation model.
[0099] Based on the preset optical flow estimation loss function, the model training unit 120 optimizes the parameters of the pre-trained optical flow estimation model to obtain the initial parameters.
[0100] After the optical flow estimation model is pre-trained, the model training unit 120 performs incremental processing on the multimodal image data generated by the multimodal optical flow data generation module.
[0101] Based on the initial parameters of the optical flow estimation model, the model training unit 120 fine-tunes the optical flow estimation model using multimodal image data that has undergone incremental processing at a preset learning rate, so as to update the parameters of the optical flow estimation model.
[0102] Model training unit 120 adjusts the parameter update frequency and learning rate of the optical flow estimation model.
[0103] During the fine-tuning of the optical flow estimation model, the model training unit 120 also inputs the incrementally processed multimodal image data into the preset neural network in the optical flow estimation model to obtain the multimodal optical flow feature prediction results.
[0104] The model training unit 120 iteratively trains the preset neural network based on the prediction results of the multimodal optical flow features to obtain the fused multimodal optical flow features.
[0105] The model training unit 120 performs feature alignment on the fused multimodal optical flow features to ensure the spatial and semantic consistency of optical flow features of different modalities.
[0106] Based on the preset feature consistency loss function, the model training unit 120 calculates the feature consistency loss of the fused multimodal optical flow features that have undergone feature alignment, in order to update the parameters of the optical flow estimation model.
[0107] The model optimization unit 130 determines the trained optical flow estimation model as the teacher model and performs feature distillation on it to obtain the intermediate layer features of the optical flow estimation model.
[0108] The model optimization unit 130 acquires a preset lightweight model to serve as the student model.
[0109] The model optimization unit 130 transfers the intermediate layer features of the optical flow estimation model to the lightweight model in order to optimize the optical flow estimation model.
[0110] Figure 3 A block diagram of an electronic device according to an example embodiment of this application is shown.
[0111] like Figure 3 As shown, the electronic device 600 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0112] like Figure 3 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the methods described in this specification according to the various exemplary embodiments of this application. For example, the processing unit 610 can perform, for example... Figure 1 The method shown.
[0113] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.
[0114] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0115] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0116] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0117] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.
[0118] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0119] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0120] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0121] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.
[0122] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.
[0123] The embodiments of this application have been described in detail above. These descriptions are solely for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, its specific implementation methods, and its application scope, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for configuring an optical flow estimation model, characterized in that, include: Multimodal image data is generated based on a preset multimodal optical flow data generation module; The preset optical flow estimation model is trained using the multimodal image data; The trained optical flow estimation model is optimized.
2. The method according to claim 1, characterized in that, Based on a pre-defined multimodal optical flow data generation module, multimodal image data is generated, including: Image source data is obtained through the multimodal optical flow data generation module; Image guidance data is obtained through the multimodal optical flow data generation module; The multimodal image data is generated by the multimodal optical flow data generation module based on the image source data and the image guidance data.
3. The method according to claim 2, characterized in that, Image source data is acquired through the multimodal optical flow data generation module, including: Obtain a pre-defined multi-view image dataset; The multi-view image dataset is preprocessed by the multimodal optical flow data generation module to obtain the image source data.
4. The method according to claim 2, characterized in that, Image guidance data is obtained through the multimodal optical flow data generation module, including: Obtain a pre-defined cross-modal image dataset; The multimodal optical flow data generation module preprocesses the cross-modal image dataset to obtain the image guidance data.
5. The method according to claim 1, characterized in that, Training a preset optical flow estimation model using the multimodal image data includes: The optical flow estimation model is pre-trained to obtain initial parameters; Based on the initial parameters, the pre-trained optical flow estimation model is fine-tuned and trained using the multimodal image data.
6. The method according to claim 5, characterized in that, Based on the initial parameters, the pre-trained optical flow estimation model is fine-tuned and feature fusion trained using the multimodal image data, including: Incremental processing is performed on the multimodal image data; Based on the initial parameters, the pre-trained optical flow estimation model is fine-tuned using incrementally processed multimodal image data at a preset learning rate to update the parameters of the optical flow estimation model. Adjust the parameter update frequency and learning rate of the optical flow estimation model.
7. The method according to claim 5, characterized in that, Based on the initial parameters, the pre-trained optical flow estimation model is fine-tuned and feature fusion trained using the multimodal image data, including: Incremental processing is performed on the multimodal image data; The multimodal image data that has undergone incremental processing is input into the preset neural network in the optical flow estimation model to obtain the multimodal optical flow feature prediction results; The preset neural network is iteratively trained based on the multimodal optical flow feature prediction results to obtain fused multimodal optical flow features; Feature alignment is performed on the fused multimodal optical flow features; Based on a preset feature consistency loss function, the feature consistency loss of the fused multimodal optical flow features that have undergone feature alignment is calculated to update the parameters of the optical flow estimation model. Adjust the parameter update frequency of the optical flow estimation model.
8. A configuration device for an optical flow estimation model, characterized in that, include: The data generation unit is used to generate multimodal image data based on a preset multimodal optical flow data generation module; The model training unit is used to train a preset optical flow estimation model using the multimodal image data; The model optimization unit is used to optimize the trained optical flow estimation model.
9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-7.