Training method of optical flow estimation model, optical flow estimation method, device and equipment

Abstract

The method comprises the following steps: extracting context features of a first image by a target encoder, and extracting multi-scale matching features of the first image and a second image respectively; constructing a first cost volume according to the matching features of the first image at a first resolution and the matching features of the second image at the first resolution; inputting the first cost volume and the context features into a first decoder to obtain a first optical flow map; upsampling the first optical flow map to a second resolution to obtain an upsampled first optical flow map, and constructing a second cost volume according to the matching features of the first image at a second resolution and the matching features of the second image at the second resolution; inputting the upsampled first optical flow map, the second cost volume and the context features into a second decoder to obtain a second optical flow map; constructing a loss function according to the second optical flow map, and adjusting model parameters of an optical flow estimation model according to the loss function.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to training methods for optical flow estimation models, optical flow estimation methods, apparatus, and devices. Background Technology

[0002] Optical flow estimation is used to determine the motion vector of each pixel between consecutive frames of a video. It is a fundamental task in computer vision for providing dense, low-level motion understanding. The output of optical flow estimation directly affects the accuracy and reliability of higher-level applications such as object tracking, motion segmentation, and scene flow computation. With the widespread adoption of real-time systems such as autonomous driving and robot navigation, the industry has placed dual demands on optical flow algorithms for high accuracy and high computational efficiency to meet the safety constraints of millisecond-level response times.

[0003] Among related technologies, architectures represented by RAFT (Recurrent All-Pairs Field Transforms) have significantly improved optical flow accuracy by constructing multi-scale feature pyramids and 4D correlators. However, subsequent improvements have generally followed the technical route of larger feature backbones and higher accuracy, which has brought significant computational costs while improving model accuracy. Summary of the Invention

[0004] This disclosure provides a training method for an optical flow estimation model, an optical flow estimation method, an apparatus, and a device.

[0005] According to a first aspect of this disclosure, a method for training an optical flow estimation model is provided, the optical flow estimation model including a target encoder, a first decoder, and a second decoder, the method comprising:

[0006] The target encoder extracts contextual features of the first image and multi-scale matching features of the first and second images respectively; wherein the first and second images are two consecutive frames, and the multi-scale matching features include at least matching features at a first resolution and matching features at a second resolution, wherein the first resolution is smaller than the second resolution;

[0007] A first cost volume is constructed based on the matching features of the first resolution of the first image and the matching features of the first resolution of the second image;

[0008] The first cost volume and the context features are input into the first decoder to obtain the first optical flow map;

[0009] The first optical flow map is upsampled to the second resolution to obtain the upsampled first optical flow map, and a second cost volume is constructed based on the matching features of the second resolution of the first image and the matching features of the second resolution of the second image.

[0010] The upsampled first optical flow map, the second cost volume, and the context features are input into the second decoder to obtain the second optical flow map;

[0011] The loss function of the optical flow estimation model is constructed based on the second optical flow map, and the model parameters of the optical flow estimation model are adjusted based on the loss function.

[0012] Optionally, the target encoder includes a first encoder and a second encoder.

[0013] The step of extracting contextual features of the first image through the target encoder, and extracting multi-scale matching features of the first and second images respectively through the target encoder, includes:

[0014] The contextual features of the first image are extracted using the first encoder; and,

[0015] The second encoder extracts multi-scale matching features from the first image and the second image, respectively.

[0016] Optionally, the context features include at least the context features of the first resolution.

[0017] The step of inputting the first cost volume and the context features into the first decoder to obtain the first optical flow map includes:

[0018] The first cost volume and the context features of the first resolution are input into the first decoder to obtain the first optical flow map.

[0019] Optionally, the context features also include context features of the second resolution.

[0020] The step of inputting the upsampled first optical flow map, the second cost volume, and the context features into the second decoder to obtain the second optical flow map includes:

[0021] The upsampled first optical flow map, the second cost volume, and the context features of the second resolution are input into the second decoder to obtain the second optical flow map.

[0022] Optionally, the loss function is a robust root scaling function.

[0023] According to a second aspect of this disclosure, an optical flow estimation method is provided, the method comprising:

[0024] Acquire the third and fourth images; wherein the third and fourth images are two consecutive frames;

[0025] The third image and the fourth image are input into the trained optical flow estimation model to obtain the third optical flow map;

[0026] The optical flow estimation model is trained according to the training method of the optical flow estimation model described in the first aspect above.

[0027] According to a third aspect of this disclosure, a training apparatus for an optical flow estimation model is provided, the optical flow estimation model including at least a target encoder, a first decoder, and a second decoder, the apparatus comprising:

[0028] The first extraction module is used to extract contextual features of the first image through the target encoder, and to extract multi-scale matching features of the first image and the second image through the target encoder respectively; wherein the first image and the second image are two consecutive frames, and the multi-scale matching features include at least matching features at a first resolution and matching features at a second resolution, wherein the first resolution is smaller than the second resolution;

[0029] The first construction module is used to construct a first cost volume based on the matching features of the first resolution of the first image and the matching features of the first resolution of the second image;

[0030] A first decoding module is used to input the first cost volume and the context features into the first decoder to obtain a first optical flow map;

[0031] The second construction module is used to upsample the first optical flow map to the second resolution to obtain the upsampled first optical flow map, and to construct a second cost volume based on the matching features of the second resolution of the first image and the matching features of the second resolution of the second image.

[0032] The second decoding module is used to input the upsampled first optical flow map, the second cost volume, and the context features into the second decoder to obtain the second optical flow map;

[0033] The adjustment module is used to construct the loss function of the optical flow estimation model based on the second optical flow map, and adjust the model parameters of the optical flow estimation model based on the loss function.

[0034] According to a fourth aspect of this disclosure, an optical flow estimation apparatus is provided, the apparatus comprising:

[0035] An acquisition module is used to acquire a third image and a fourth image; wherein the third image and the fourth image are two consecutive frames;

[0036] The estimation module is used to input the third image and the fourth image into the trained optical flow estimation model to obtain the third optical flow map;

[0037] The optical flow estimation model is trained according to the training method of the optical flow estimation model described in the first aspect above.

[0038] According to a fifth aspect of this disclosure, an electronic device is provided, including a memory for storing executable computer instructions; and a processor for executing, under the control of the executable computer instructions, a training method for an optical flow estimation model according to the first aspect above or an optical flow estimation method according to the second aspect above.

[0039] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, perform the training method for the optical flow estimation model described in the first aspect above or the optical flow estimation method described in the second aspect above.

[0040] According to the training method of the optical flow estimation model in the embodiments of this disclosure, the training process of the optical flow estimation model adopts a training strategy of multi-scale matching feature extraction, full fusion of context features, and iterative optimization from coarse to fine, so that the optical flow estimation model can maintain high accuracy while significantly reducing the computational complexity of the model.

[0041] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0043] Figure 1 This is a schematic diagram of the hardware configuration of an electronic device provided according to an embodiment of the present disclosure;

[0044] Figure 2 This is a flowchart illustrating the training method for the optical flow estimation model provided in the embodiments of this disclosure;

[0045] Figure 3 This is a schematic diagram of the training process of the optical flow estimation model provided according to the embodiments of this disclosure;

[0046] Figure 4 This is a flowchart illustrating the optical flow estimation method provided according to embodiments of this disclosure;

[0047] Figure 5This is a schematic block diagram of a training apparatus for an optical flow estimation model provided according to an embodiment of the present disclosure;

[0048] Figure 6 This is a schematic block diagram of an optical flow estimation device provided according to an embodiment of the present disclosure;

[0049] Figure 7 This is a schematic diagram of the hardware configuration of another electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0050] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0051] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0052] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0053] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0054] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0055] <Hardware Configuration>

[0056] Figure 1 This is a block diagram of the hardware configuration of an electronic device 1000 according to an embodiment of the present disclosure.

[0057] In one embodiment, the electronic device 1000 may be a server or a terminal device. The server may be a monolithic server or a distributed server spanning multiple computers or a computer data center. The terminal device may be a portable computer, desktop computer, wearable device, or any other device having a processor or other computing device and a memory or other storage device; this embodiment does not limit the specific device.

[0058] like Figure 1As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc.

[0059] Processor 1100 may be a mobile processor. Memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. Interface device 1300 includes, for example, a USB interface and a headphone jack. Communication device 1400 may be capable of wired or wireless communication. Communication device 1400 may include short-range communication devices, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. Communication device 1400 may also include long-range communication devices, such as any device that performs WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication. Display device 1500 is, for example, an LCD screen or a touch screen. Input device 1600 may include, for example, a touch screen or a keyboard. Electronic device 1000 can output audio information through a speaker 1700 and acquire audio information through a microphone 1800.

[0060] Despite Figure 1 The electronic device 1000 shows multiple devices, but this disclosure may only relate to some of them. For example, electronic device 1000 may only relate to memory 1200 and processor 1100.

[0061] In embodiments of this disclosure, the memory 1200 of the electronic device 1000 is used to store instructions for controlling the processor 1100 to execute the training method and optical flow estimation method of the optical flow estimation model provided in the embodiments of this disclosure.

[0062] In the above description, those skilled in the art can design instructions based on the scheme disclosed in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0063] <Method Example 1>

[0064] In this embodiment, a training method for an optical flow estimation model is provided. This training method can be implemented by an electronic device. The optical flow estimation model includes a target encoder, a first decoder, and a second decoder. The electronic device can be, for example, […]. Figure 1 The electronic device shown is 1000.

[0065] according to Figure 2As shown, the training method of the optical flow estimation model in this embodiment may include the following steps S2100 to S2500.

[0066] Step S2100: Extract contextual features of the first image through the target encoder, and extract multi-scale matching features of the first image and the second image respectively through the target encoder.

[0067] Wherein, the first image and the second image are two consecutive frames. For example, the first image and the second image can be two adjacent consecutive frames in a video, and the first image is located before the second image. The first image can be the starting frame image, which can be called the source image, and the second image can be the ending frame image, which can be called the target image. The purpose of optical flow is to find the motion trajectory from the starting point to the ending point.

[0068] The multi-scale matching features include at least matching features at a first resolution and matching features at a second resolution, where the first resolution is smaller than the second resolution. The context features include at least context features at the first resolution and context features at the second resolution, and these context features can serve as multi-scale context features. For example, the first resolution can be 1 / 32 of the original image resolution, and the second resolution can be 1 / 16 of the original image resolution. In this case, the context features can include context features at 1 / 32 of the original image resolution and context features at 1 / 16 of the original image resolution.

[0069] In one example, the target encoder can be an edge motion segmentation encoder, which may include a first encoder and a second encoder. The first encoder can be a contextual feature encoder, used to extract contextual features of the image. Contextual features typically include scene, semantic, and structural information of the image (e.g., object edges, shapes), which can guide the decoder to generate a more reasonable and smoother optical flow field. The second encoder can be a multi-scale feature encoder with shared weights, used to extract matching features of the image at different resolutions.

[0070] In one embodiment of this disclosure, step S2100, which involves extracting contextual features of the first image through the target encoder and extracting multi-scale matching features of the first and second images respectively through the target encoder, may further include: extracting contextual features of the first image through the first encoder; and extracting multi-scale matching features of the first and second images respectively through the second encoder.

[0071] In one example, multi-scale contextual features of a first image can be extracted using a first encoder, and multi-scale matching features of the first image and the second image can be extracted using a second encoder.

[0072] Continuing with the example above, refer to... Figure 3 The first image is input into the first encoder of the optical flow estimation model. The first encoder extracts multi-scale contextual features of the first image, such as contextual features V1 at 1 / 32 of the original resolution and contextual features V2 at 1 / 16 of the original resolution. The first and second images are then input into a second encoder, which extracts multi-scale matching features of both the first and second images. The multi-scale matching features of the first image may include, for example, matching features C11 and C12 at 1 / 32 of the original resolution and matching features C21 and C22 at 1 / 16 of the original resolution.

[0073] Step S2200: Construct a first cost volume based on the matching features of the first resolution of the first image and the matching features of the first resolution of the second image.

[0074] In this embodiment, a first cost volume can be constructed based on the matching features of the first resolution of the first image and the matching features of the second image at the first resolution. Typically, the cost volume is a four-dimensional data structure that densely records the matching cost between each point in the input image and each possible candidate point in the input image.

[0075] Continuing with the example above, refer to... Figure 3 The first cost volume CV1 can be constructed based on the matching features C11 and C21.

[0076] Step S2300: Input the first cost volume and the context features into the first decoder to obtain the first optical flow map.

[0077] In one example, the context features include context features at a first resolution. The specific implementation of inputting the first cost volume and context features into the first decoder to obtain the first optical flow map may include: inputting the first cost volume and context features at the first resolution into the first decoder to obtain the first optical flow map.

[0078] Continuing with the example above, refer to... Figure 3The context feature V1 and the first cost volume CV1 can be input into the first decoder of the optical flow estimation model to obtain the first optical flow map.

[0079] Step S2400: Upsample the first optical flow map to the second resolution to obtain the upsampled first optical flow map, and construct the second cost volume based on the matching features of the second resolution of the first image and the matching features of the second resolution of the second image.

[0080] Continuing with the example above, the first optical flow map can be upsampled to 1 / 16 of its resolution to obtain the upsampled first optical flow map, and the matching features C12 and C22 can be used to construct the second cost volume CV2.

[0081] Step S2500: Input the upsampled first optical flow map, the second cost volume, and the context features into the second decoder to obtain the second optical flow map.

[0082] In one example, the context features include context features at a second resolution. The specific implementation of inputting the upsampled first optical flow map, the second cost volume, and the context features into the second encoder to obtain the second optical flow map may include: inputting the upsampled first optical flow map, the second cost volume, and the context features at a second resolution into the second decoder to obtain the second optical flow map.

[0083] Continuing with the example above, the upsampled first optical flow map, contextual features V2, and second cost volume CV2 can be input into the second decoder of the optical flow estimation model to obtain the second optical flow map.

[0084] It should be noted that the output optical flow usually needs to be put back into the decoder and output again (replacing the low-resolution upsampled optical flow as part of the input), and this process is repeated. This allows for gradual optimization from generating a coarse optical flow to a fine optical flow.

[0085] Step S2600: Construct the loss function of the optical flow estimation model based on the second optical flow map, and adjust the model parameters of the optical flow estimation model based on the loss function.

[0086] The loss function can be a robust root scaler function, which is not only robust to external outliers but also effectively captures subtle errors.

[0087] According to the training method of the optical flow estimation model in the embodiments of this disclosure, the training process of the optical flow estimation model adopts a training strategy of multi-scale matching feature extraction, full fusion of context features, and iterative optimization from coarse to fine, so that the optical flow estimation model can maintain high accuracy while significantly reducing the computational complexity of the model.

[0088] In one embodiment of this disclosure, the first decoder and the second decoder may employ an improved ConvNeXt module, which removes LayerNorm and LayerScale from the standard ConvNeXt and replaces the 1×1 convolutions at the end of the MLP with standard 3×3 convolutions to expand the receptive field and reduce computational overhead.

[0089] Typically, the basic ConvNeXt architecture includes convolution processing, normalization processing, and 1×1 convolution processing, with each layer having a pre-layerNorm operation and a post-layerScale operation. The decoder in this embodiment removes the LayerNorm and LayerScale operations from the basic ConvNeXt architecture, reducing computational overhead. Furthermore, the decoder in this embodiment replaces the 1×1 convolutions at the ends of the MLP with standard 3×3 convolutions, improving the decoder's receptive field and accuracy.

[0090] In one embodiment of this disclosure, the optical flow estimation model further includes a third encoder, and the multi-scale matching features may further include matching features at a third resolution, and the context features may further include context features at a third resolution, wherein the third resolution is smaller than the second resolution. The training method for the optical flow estimation model in this embodiment may further include: upsampling the second optical flow map to a third resolution to obtain an upsampled second optical flow map; constructing a third cost volume based on the matching features at the third resolution of the first image and the matching features at the third resolution of the second image; inputting the upsampled second optical flow map, the third cost volume, and the context features at the third resolution of the first image into the third encoder to obtain a third optical flow map; and constructing a loss function for the optical flow estimation model based on the third optical flow map, and adjusting the model parameters of the optical flow estimation model based on the loss function.

[0091] For example, the first encoder also extracts contextual features V3 at 1 / 8 resolution of the original resolution of the first image, and the second encoder also extracts matching features C13 and C23 at 1 / 8 resolution of the original resolution of the first image. The second optical flow map can be upsampled to 1 / 8 resolution to obtain the upsampled second optical flow map, and a third cost volume CV3 can be constructed based on matching features C13 and C23. Further, the upsampled second optical flow map, contextual features V3, and second cost volume CV3 can be input into the third decoder of the optical flow estimation model to obtain the third optical flow map. A loss function for the optical flow estimation model is then constructed based on the third optical flow map, and the model parameters of the optical flow estimation model are adjusted according to the loss function.

[0092] <Method Example 2>

[0093] In this embodiment, an optical flow estimation method is provided. This optical flow estimation method can be implemented by an electronic device, which can be, for example, such as... Figure 1 The electronic device shown is 1000.

[0094] according to Figure 4 As shown, the optical flow estimation method of this disclosure embodiment may include the following steps S4100 to S4200.

[0095] Step S4100: Obtain the third and fourth images.

[0096] The third and fourth images are two consecutive frames. Typically, the third and fourth images are two adjacent consecutive frames in a video, with the third image preceding the fourth image.

[0097] Step S4200: Input the third image and the fourth image into the trained optical flow estimation model to obtain the third optical flow map;

[0098] The optical flow estimation model is trained using the training method described in Embodiment 1 above.

[0099] In this embodiment, the third and fourth images are input into a pre-trained optical flow estimation model to obtain the third optical flow map.

[0100] According to the optical flow estimation method of this disclosure, the optical flow estimation model adopts a training strategy of multi-scale matching feature extraction, full fusion of context features, and iterative optimization from coarse to fine during the training process, which enables the optical flow estimation model to maintain high accuracy while significantly reducing the computational complexity of the model.

[0101] <Device Example 1>

[0102] In this embodiment, a training device 5000 for an optical flow estimation model is provided. The optical flow estimation model includes at least a target encoder, a first decoder, and a second decoder, such as... Figure 5 As shown, the training device 5000 for the optical flow estimation model may include a first extraction module 5100, a first construction module 5200, a first decoding module 5300, a second construction module 5400, a second decoding module 5500, and an adjustment module 5600.

[0103] The first extraction module 5100 is used to extract contextual features of the first image through the target encoder, and to extract multi-scale matching features of the first image and the second image through the target encoder respectively; wherein the first image and the second image are two consecutive frames, and the multi-scale matching features include at least matching features at a first resolution and matching features at a second resolution, wherein the first resolution is smaller than the second resolution;

[0104] The first construction module 5200 is used to construct a first cost volume based on the matching features of the first resolution of the first image and the matching features of the first resolution of the second image;

[0105] The first decoding module 5300 is used to input the first cost volume and the context features into the first decoder to obtain the first optical flow map;

[0106] The second construction module 5400 is used to upsample the first optical flow map to the second resolution to obtain the upsampled first optical flow map, and to construct a second cost volume based on the matching features of the second resolution of the first image and the matching features of the second resolution of the second image.

[0107] The second decoding module 5500 is used to input the upsampled first optical flow map, the second cost volume and the context features into the second decoder to obtain the second optical flow map;

[0108] The adjustment module 5600 is used to construct the loss function of the optical flow estimation model based on the second optical flow map, and adjust the model parameters of the optical flow estimation model based on the loss function.

[0109] In one embodiment, the target encoder includes a first encoder and a second encoder.

[0110] The first extraction module 5100 is specifically used to extract the context features of the first image through the first encoder; and to extract the multi-scale matching features of the first image and the second image through the second encoder respectively.

[0111] In one embodiment, the context features include at least the context features of the first resolution.

[0112] The first decoding module 5300 is specifically used to input the first cost volume and the context features of the first resolution into the first decoder to obtain the first optical flow map.

[0113] In one embodiment, the context features include at least the context features of the second resolution.

[0114] The second decoding module 5400 is specifically used to input the upsampled first optical flow map, the second cost volume, and the context features of the second resolution into the second decoder to obtain the second optical flow map.

[0115] In one embodiment, the upsampled first optical flow map, the second cost volume, and the context features of the second resolution are input into the second decoder to obtain the second optical flow map.

[0116] According to this embodiment, during the training process of its optical flow estimation model, a training strategy of multi-scale matching feature extraction, full-process fusion of context features, and iterative optimization from coarse to fine is adopted, which enables the optical flow estimation model to maintain high accuracy while significantly reducing the computational complexity of the model.

[0117] <Device Embodiment Two>

[0118] In this embodiment, an optical flow estimation device 6000 is provided, such as... Figure 6 As shown, the optical flow estimation device 6000 may include an acquisition module 6100 and an estimation module 6200.

[0119] The acquisition module 6100 is used to acquire a third image and a fourth image; wherein the third image and the fourth image are two consecutive frames;

[0120] The estimation module 6200 is used to input the third image and the fourth image into the trained optical flow estimation model to obtain the third optical flow map;

[0121] The optical flow estimation model is trained using the training method described in Embodiment 1 above.

[0122] According to this embodiment, because the optical flow estimation model employs a training strategy of multi-scale matching feature extraction, full-process fusion of contextual features, and iterative optimization from coarse to fine during training, the optical flow estimation model maintains high accuracy while significantly reducing the computational complexity of the model.

[0123] <Equipment Example>

[0124] Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to one embodiment. For example... Figure 7 As shown, the electronic device 7000 includes a processor 7100 and a memory 7200.

[0125] The memory 7200 can be used to store executable computer instructions.

[0126] The processor 7100 can be used to execute, under the control of the executable computer instructions, the training method of the optical flow estimation model according to Embodiment 1 of the present disclosure or the optical flow estimation method according to Embodiment 2 of the present disclosure.

[0127] The electronic device 7000 can be as follows: Figure 1 The electronic device 1000 shown may also be a device with other hardware structures, which are not limited here.

[0128] In another embodiment, the electronic device 7000 may include a training device 5000 or an optical flow estimation device 6000 for the above optical flow estimation model.

[0129] In one embodiment, the training device 5000 or the modules of the optical flow estimation device 6000 can be implemented by the processor 7100 running computer instructions stored in the memory 7200.

[0130] Computer-readable storage media

[0131] This disclosure also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, perform either the training method for the optical flow estimation model provided in Embodiment 1 of this disclosure or the optical flow estimation method provided in Embodiment 2 of this disclosure.

[0132] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0133] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0134] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0135] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0136] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0137] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0138] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0140] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A training method for an optical flow estimation model, characterized in that, The optical flow estimation model includes a target encoder, a first decoder, and a second decoder. The target encoder includes a first encoder and a second encoder. The first decoder and the second decoder employ an improved ConvNeXt module. The improved ConvNeXt module removes LayerNorm and LayerScale from the standard ConvNeXt module and replaces the 1×1 convolutions at the ends of the MLP with standard 3×3 convolutions. The method includes: The target encoder extracts contextual features of the first image and multi-scale matching features of the first and second images respectively; wherein the first and second images are two consecutive frames, and the multi-scale matching features include at least matching features at a first resolution and matching features at a second resolution, wherein the first resolution is smaller than the second resolution; the first image is located before the second image, the first image is the starting frame image, and the second image is the ending frame image, and the purpose of optical flow is to find the motion trajectory from the starting point to the ending point; A first cost volume is constructed based on the matching features of the first resolution of the first image and the matching features of the first resolution of the second image; The first cost volume and the context features are input into the first decoder to obtain the first optical flow map; The first optical flow map is upsampled to the second resolution to obtain the upsampled first optical flow map, and a second cost volume is constructed based on the matching features of the second resolution of the first image and the matching features of the second resolution of the second image. The upsampled first optical flow map, the second cost volume, and the context features are input into the second decoder to obtain the second optical flow map; The loss function of the optical flow estimation model is constructed based on the second optical flow map, and the model parameters of the optical flow estimation model are adjusted based on the loss function; The steps of extracting contextual features of the first image through the target encoder and extracting multi-scale matching features of the first and second images respectively through the target encoder include: The first image is input into the first encoder, and the multi-scale context features of the first image are extracted by the first encoder. The multi-scale context features include context features V1 at 1 / 32 of the original resolution of the first image and context features V2 at 1 / 16 of the original resolution of the first image. The first image and the second image are input into the second encoder. The second encoder extracts the multi-scale matching features of the first image and the second image. The multi-scale matching features of the first image include a matching feature C11 at 1 / 32 of the original resolution of the first image and a matching feature C12 at 1 / 16 of the original resolution of the first image. The multi-scale matching features of the second image include a matching feature C21 at 1 / 32 of the original resolution of the second image and a matching feature C22 at 1 / 16 of the original resolution of the second image.

2. The method according to claim 1, characterized in that, The context features include at least the context features of the first resolution. The step of inputting the first cost volume and the context features into the first decoder to obtain the first optical flow map includes: The first cost volume and the context features of the first resolution are input into the first decoder to obtain the first optical flow map.

3. The method according to claim 1, characterized in that, The context features also include the context features of the second resolution. The step of inputting the upsampled first optical flow map, the second cost volume, and the context features into the second decoder to obtain the second optical flow map includes: The upsampled first optical flow map, the second cost volume, and the context features of the second resolution are input into the second decoder to obtain the second optical flow map.

4. The method according to claim 1, characterized in that, The loss function is a robust root scaling function.

5. An optical flow estimation method, characterized in that, The method includes: Acquire the third and fourth images; wherein the third and fourth images are two consecutive frames; The third image and the fourth image are input into the trained optical flow estimation model to obtain the third optical flow map; The optical flow estimation model is trained using the training method for the optical flow estimation model according to any one of claims 1 to 4.

6. A training device for an optical flow estimation model, characterized in that, The optical flow estimation model includes at least a target encoder, a first decoder, and a second decoder. The target encoder includes a first encoder and a second encoder. The first decoder and the second decoder employ an improved ConvNeXt module. The improved ConvNeXt module removes LayerNorm and LayerScale from the standard ConvNeXt module and replaces the 1×1 convolutions at the ends of the MLP with standard 3×3 convolutions. The device includes: The first extraction module is used to extract contextual features of the first image through the target encoder, and to extract multi-scale matching features of the first image and the second image respectively through the target encoder; wherein the first image and the second image are two consecutive frames, and the multi-scale matching features include at least matching features at a first resolution and matching features at a second resolution, wherein the first resolution is smaller than the second resolution; the first image is located before the second image, the first image is the starting frame image, and the second image is the ending frame image, and the purpose of optical flow is to find the motion trajectory from the starting point to the ending point; The first construction module is used to construct a first cost volume based on the matching features of the first resolution of the first image and the matching features of the first resolution of the second image; A first decoding module is used to input the first cost volume and the context features into the first decoder to obtain a first optical flow map; The second construction module is used to upsample the first optical flow map to the second resolution to obtain the upsampled first optical flow map, and to construct a second cost volume based on the matching features of the second resolution of the first image and the matching features of the second resolution of the second image. The second decoding module is used to input the upsampled first optical flow map, the second cost volume, and the context features into the second decoder to obtain the second optical flow map; An adjustment module is used to construct a loss function for the optical flow estimation model based on the second optical flow map, and to adjust the model parameters of the optical flow estimation model based on the loss function; Specifically, the first extraction module is used to: input the first image into the first encoder, and extract multi-scale context features of the first image through the first encoder. The multi-scale context features include context features V1 at 1 / 32 resolution of the original resolution of the first image and context features V2 at 1 / 16 resolution of the original resolution of the first image. The first image and the second image are input into the second encoder. The second encoder extracts the multi-scale matching features of the first image and the second image. The multi-scale matching features of the first image include a matching feature C11 at 1 / 32 of the original resolution of the first image and a matching feature C12 at 1 / 16 of the original resolution of the first image. The multi-scale matching features of the second image include a matching feature C21 at 1 / 32 of the original resolution of the second image and a matching feature C22 at 1 / 16 of the original resolution of the second image.

7. An optical flow estimation device, characterized in that, The device includes: An acquisition module is used to acquire a third image and a fourth image; wherein the third image and the fourth image are two consecutive frames; The estimation module is used to input the third image and the fourth image into the trained optical flow estimation model to obtain the third optical flow map; The optical flow estimation model is trained using the training method for the optical flow estimation model according to any one of claims 1 to 4.

8. An electronic device, characterized in that, include: Memory is used to store executable computer instructions; A processor, configured to execute, under the control of the executable computer instructions, the training method of the optical flow estimation model according to any one of claims 1 to 4 or the optical flow estimation method according to claim 5.

9. A computer-readable storage medium having computer instructions stored thereon, the computer instructions being executed by a processor to perform a training method for an optical flow estimation model according to any one of claims 1 to 4 or an optical flow estimation method according to claim 5.

Images