Image optical flow estimation method, device, equipment and medium
By acquiring the source image and target image with reduced number of channels, determining the similarity of pixel points and generating costs, the lightweight CNN network structure solves the problems of complex calculation of image optical flow information and resource consumption, and realizes efficient optical flow information recognition.
Patent Information
- Application Number
- CN202411109932.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2025-07-25
AI Technical Summary
In the prior art, the identification and calculation of image optical flow information is complex and consumes computing resources, and is not efficient.
By acquiring the source image and the target image with reduced number of channels, the similarity between pixel points is determined, the cost amount is generated, and the optical flow information is determined based on the cost amount, and the feature extraction and encoding is performed using the lightweight CNN network structure.
It reduces the computational complexity, saves computing resources, and improves the recognition efficiency and accuracy of optical flow information.
Smart Images
Figure CN120374493A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular, to an image optical flow estimation method, apparatus, device, and medium. Background Art
[0002] Burst image processing mainly belongs to the research in the fields of computer vision and image processing, especially in aspects such as multi-frame image synthesis, image enhancement, super-resolution reconstruction, and improvement of image quality under low-light conditions. Burst image processing refers to analyzing and processing an image sequence captured quickly and continuously by a device, and using an algorithm to select the best image or perform composite processing on all images to obtain a high-quality image output. The optical flow image alignment technology is an important technology in the field of computer vision for estimating the motion information of objects in an image sequence. By analyzing the pixel-level changes between consecutive frames to determine the motion of objects, it is widely used in multiple fields such as motion detection, target tracking, 3D reconstruction, and image registration. During the optical flow image alignment process, it is necessary to identify the optical flow information of each pixel point in the image.
[0003] In related technologies, the identification and calculation of optical flow information in an image are complex, consume computing resources, and have low efficiency. Summary of the Invention
[0004] The present disclosure aims to at least solve one of the technical problems in the related technologies to some extent.
[0005] To this end, the present disclosure proposes an image optical flow estimation method, apparatus, electronic device, computer-readable storage medium, and computer program product to reduce the computational complexity, save computing resources, and improve efficiency.
[0006] In a first aspect of an embodiment of the present disclosure, an image optical flow estimation method is proposed, including: obtaining a source image corresponding to a first image, and obtaining a target image corresponding to a second image, where the number of channels of the source image is less than the number of channels of the first image, the number of channels of the target image is less than the number of channels of the second image, and the acquisition time of the second image is after the acquisition time of the first image; determining the similarity between each first pixel point in the source image and each second pixel point in the target image; generating a cost volume of the source image, where the cost volume includes the similarity corresponding to each first pixel point; and determining the optical flow information of each first pixel point according to the cost volume of the source image.
[0007] In a second aspect embodiment of the present disclosure, an image optical flow estimation device is provided, including: an acquisition module, configured to acquire a source image corresponding to a first image and acquire a target image corresponding to a second image, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after that of the first image; a first determination module, configured to determine the similarity between each first pixel point in the source image and each second pixel point in the target image; a generation module, configured to generate a cost volume of the source image, where the cost volume includes the similarity corresponding to each first pixel point; and a second determination module, configured to determine the optical flow information of each first pixel point according to the cost volume of the source image.
[0008] In a third aspect embodiment of the present disclosure, an electronic device is provided, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the image optical flow estimation method provided in the first aspect embodiment of the present disclosure.
[0009] In a fourth aspect embodiment of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the image optical flow estimation method provided in the first aspect embodiment of the present disclosure.
[0010] In a fifth aspect embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, it implements the image optical flow estimation method provided in the first aspect embodiment of the present disclosure.
[0011] The image optical flow estimation method, device, electronic device, computer-readable storage medium, and computer program product provided by the present disclosure acquire a source image corresponding to a first image and acquire a target image corresponding to a second image, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after that of the first image, determine the similarity between each first pixel point in the source image and each second pixel point in the target image, generate a cost volume of the source image, where the cost volume includes the similarity corresponding to each first pixel point, and determine the optical flow information of each first pixel point according to the cost volume of the source image, so as to reduce the computational complexity, save computing resources, and improve efficiency.
[0012] Additional aspects and advantages of the present disclosure will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present disclosure. Description of the Drawings
[0013] The above and / or additional aspects and advantages of the present disclosure will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:
[0014] Figure 1 It is a schematic flowchart of an image optical flow estimation method provided by an embodiment of the present disclosure;
[0015] Figure 2 It is a schematic flowchart of another image optical flow estimation method provided by an embodiment of the present disclosure;
[0016] Figure 3 It is an application schematic diagram in an embodiment of the present disclosure;
[0017] Figure 4 It is a schematic structural diagram of an image optical flow estimation device provided by an embodiment of the present disclosure;
[0018] Figure 5 It shows a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present disclosure;
[0019] Figure 6 It is a schematic structural diagram of a chip proposed by an embodiment of the present disclosure. Detailed Embodiments
[0020] Here, some embodiments of the present disclosure will be described in detail, and their examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those set forth herein, but can be changed as will be apparent after understanding the present disclosure, except for operations that must be performed in a specific order. Additionally, descriptions of features known in the art may be omitted for the sake of clarity and conciseness.
[0021] The embodiments described in some embodiments of the present disclosure below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0022] It should be noted that all actions of obtaining signals, information, or data in the present disclosure are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and obtaining authorization from the owner of the corresponding device.
[0023] Figure 1 It is a schematic flowchart of an image optical flow estimation method provided by an embodiment of the present disclosure.
[0024] In this embodiment, the image optical flow estimation method is configured as an image optical flow estimation device for illustration. In this embodiment, the image optical flow estimation method can be configured in the image optical flow estimation device, and the image optical flow estimation device can be set in the server or in the electronic device, and there is no limitation on this.
[0025] This embodiment takes the image optical flow estimation method configured in the electronic device as an example. Among them, the electronic device is a hardware device with various operating systems such as a smart phone, a tablet computer, a personal digital assistant, an e-book, etc.
[0026] It should be noted that the execution subject of the embodiments of the present disclosure can be, for example, a central processing unit (CPU) in a server or an electronic device in terms of hardware, and can be, for example, a relevant background service in a server or an electronic device in terms of software, and there is no limitation on this.
[0027] As Figure 1 shown, the image optical flow estimation method includes:
[0028] Step S101, obtain a source image corresponding to the first image and obtain a target image corresponding to the second image, where the number of channels of the source image is less than the number of channels of the first image, the number of channels of the target image is less than the number of channels of the second image, and the acquisition time of the second image is after the acquisition time of the first image.
[0029] In some embodiments, images obtained by the electronic device continuously shooting a reference object can be used as the first image and the second image, or the electronic device can receive the first image and the second image captured and sent by other devices, and there is no limitation on this.
[0030] Among them, the acquisition time of the second image being after the acquisition time of the first image means that the second image is an image of the reference object taken after the first image.
[0031] In some embodiments, the first image and the second image can be a pair of burst images (assumed to be n) obtained by the camera taking consecutive shots. The first image can be the first frame image among the n images, and the second image can be the second frame, …, the nth frame image, and there is no limitation on this.
[0032] After the above-mentioned first image and second image of the reference object are acquired, the optical flow information of each first pixel point (the pixel points in the first image can be called first pixel points) in the first image can be calculated. The optical flow information can characterize the displacement situation (which can also be called the displacement field) of each first pixel point in at least one subsequent acquired second image.
[0033] The number of the above-mentioned first images can be one, and the number of the second images can be one or more, without any limitation in this regard.
[0034] Among them, the number of source images can be one, and the number of target images can be one or more, without any limitation in this regard.
[0035] After acquiring the first image and the second image of the reference object, the source image corresponding to the first image and the target image corresponding to the second image can be obtained. Among them, the number of channels of the source image is less than that of the first image, and the number of channels of the target image is less than that of the second image. Since the number of channels of the first image and the second image is reduced, the data volume is reduced, and the delay caused by data transfer is supported to be reduced, so as to overall support reducing the computational complexity, saving computing resources, and improving efficiency.
[0036] In some embodiments, the image formats of the first image and the second image are RGB formats; then in the process of implementing the acquisition of the source image corresponding to the first image and the acquisition of the target image corresponding to the second image, the RAW image corresponding to the first image output by the image sensor can be used as the source image, and the RAW image corresponding to the second image output by the image sensor can be used as the target image. This supports quickly improving the reduction efficiency and effect of the number of channels, supports reducing the delay caused by data transfer, so as to overall support reducing the computational complexity, saving computing resources, and improving efficiency.
[0037] Among them, the first image and the second image obtained during the imaging process can be RGB images
[0038] Among them, the RGB format is a color representation method, which is often used in computer graphics and digital image processing. RGB represents Red, Green, and Blue. These three colors can generate various other colors through different mixing ratios. In the RGB mode, the value range of each color is usually from 0 to 255, where 0 means no component of this color, and 255 means the maximum intensity of this color component. The RAW format usually refers to an uncompressed or unprocessed digital image file format. This format directly records the original image data captured from the camera sensor, and is usually encoded with the actual data collected by each photosensitive element (such as the pixels of the camera sensor), without applying any image processing or compression algorithms.
[0039] Step S102, determine the similarity between each first pixel point in the source image and each second pixel point in the target image.
[0040] After obtaining the source image corresponding to the first image and the target image corresponding to the second image, the similarity between each first pixel point in the source image and each second pixel point in the target image is determined. Among them, the pixel points in the second image can be referred to as second pixel points.
[0041] Exemplarily, for the first pixel points in the first image, the similarity between each of them and each second pixel point in the second image can be calculated (for example, obtained by vector inner product calculation), obtaining multiple similarities corresponding to the first pixel point, and performing the foregoing calculation for each first pixel point, to obtain the similarity between each first pixel point and each second pixel point in the target image.
[0042] Optionally, in some embodiments, in the process of determining the similarity between each first pixel point in the source image and each second pixel point in the target image, the source image may be processed based on a first network model to obtain a source feature map, the target image may be processed based on a second network model to obtain a target feature map, the first feature information of the first pixel point may be obtained from the source feature map, the second feature information of each second pixel point may be obtained from the target feature map, and the similarity between the first pixel point and each second pixel point may be determined according to the first feature information and each second feature information. Thereby, the extraction efficiency and effect of the similarity between the first pixel point and each second pixel point are improved.
[0043] Among them, the structure of the first network model is a Convolutional Neural Networks (CNN) network structure, and the structure of the second network model is a CNN network structure. The first network model and the second network model may be two models. The first network model is used to extract the source feature map of the source image (including the image features of the source image), and the second network model is used to extract the source feature map of the target image (including the image features of the target image).
[0044] Therefore, since the CNN network structure is a lightweight network structure and is convenient for deployment, the source feature map is extracted based on the first network model with the CNN network structure, and the target feature map is extracted based on the second network model with the CNN network structure, which can be conveniently deployed and implemented on mobile devices, effectively expand the application scenarios, and improve the practicality.
[0045] Step S103, generating a cost volume of the source image, where the cost volume includes the similarities corresponding to each first pixel point.
[0046] After determining the similarity between each first pixel point in the source image and each second pixel point in the target image as described above, a cost volume of the source image can be generated. Here, the cost volume can be expressed as cost volume (or 4D cost volume), and the cost volume can also be referred to as cost volume. The cost volume is used to identify the similarity corresponding to each first pixel point.
[0047] Step S104: Determine the optical flow information of each first pixel point according to the cost volume of the source image.
[0048] After generating the cost volume of the source image as described above, the optical flow information of each first pixel point can be determined based on the cost volume of the source image.
[0049] In some embodiments, the cost volume of the source image can be processed in combination with artificial intelligence AI to determine the optical flow information of each first pixel point; or, the cost volume of the source image can be processed based on an annotation method to determine the optical flow information of each first pixel point; or any other possible method can be used to process the cost volume of the source image to obtain the optical flow information of each first pixel point, and no limitation is imposed on this.
[0050] In this embodiment, by obtaining the source image corresponding to the first image and obtaining the target image corresponding to the second image, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after the acquisition time of the first image, determine the similarity between each first pixel point in the source image and each second pixel point in the target image, generate the cost volume of the source image, where the cost volume includes the similarity corresponding to each first pixel point, and determine the optical flow information of each first pixel point according to the cost volume of the source image. To achieve reducing the computational complexity, saving computing resources, and improving efficiency.
[0051] Figure 2 It is a schematic flowchart of another image optical flow estimation method provided by an embodiment of the present disclosure.
[0052] As Figure 2 shown, the image optical flow estimation method includes:
[0053] Step S201: Obtain the source image corresponding to the first image and obtain the target image corresponding to the second image, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after the acquisition time of the first image.
[0054] Step S202: Determine the similarity between each first pixel point in the source image and each second pixel point in the target image.
[0055] For the descriptions of steps S201 - S202, reference can be specifically made to the above - mentioned embodiments, which will not be elaborated here.
[0056] Step S203: Generate a cost map corresponding to the first pixel point according to the similarity corresponding to the first pixel point.
[0057] Among them, the cost map can be used to represent the similarity between a first pixel point and each second pixel point. The cost map corresponds to the first pixel point, that is, each first pixel point corresponds to a cost map.
[0058] Step S204: Generate a cost volume of the source image according to multiple cost maps respectively corresponding to multiple first pixel points, where the cost volume includes the similarity corresponding to each first pixel point.
[0059] After generating the cost map corresponding to the first pixel point according to the similarity corresponding to the first pixel point, the cost volume of the source image can be generated according to multiple cost maps respectively corresponding to multiple first pixel points. For example, multiple cost maps can be integrated to obtain the cost volume. During the process of generating the cost volume, the first pixel points corresponding to each cost map can be marked, and there is no limitation on this.
[0060] Thus, by generating a cost map corresponding to the first pixel point according to the similarity corresponding to the first pixel point, and generating a cost volume of the source image according to multiple cost maps respectively corresponding to multiple first pixel points, where the cost volume includes the similarity corresponding to each first pixel point, it can support efficient encoding processing of the cost volume of the source image in the subsequent process, and improve the recognition efficiency and recognition effect of optical flow information.
[0061] Step S205: Encode the cost volume of the source image to obtain cost memory information.
[0062] After obtaining the cost volume of the source image, the cost volume of the source image can be encoded to obtain cost memory information. Among them, the cost memory information can be expressed as cost memory, and the cost memory information can also be referred to as cost storage, cost memory, etc., and there is no limitation on this. The cost memory information represents the result characteristics obtained by encoding the cost volume of the source image. By encoding the cost volume of the source image to obtain cost memory information, it can support effective recognition and processing of the visual similarity information carried therein.
[0063] Exemplarily, a cost memory encoder can be used to encode the cost volume of the source image to obtain cost memory information. The process of the cost memory encoder encoding the cost volume can include: cost map patching, cost patch embedding, cost memory encoding, and there is no limitation on this.
[0064] Step S206: Determine the optical flow information of each first pixel point based on the context features and cost memory information of each first pixel point in the source image.
[0065] After encoding the cost volume of the source image to obtain the cost memory information, the optical flow information of each first pixel point can be determined by combining the context features and cost memory information of each first pixel point in the source image. This can improve the estimation accuracy and efficiency of the optical flow information.
[0066] Optionally, in some embodiments, in the process of implementing the determination of the optical flow information of each first pixel point based on the context features and cost memory information of each first pixel point in the source image, the cost features of each first pixel point can be retrieved from the cost memory information, and the optical flow information of each first pixel point can be predicted based on the cost features and context features of each first pixel point. This can greatly improve the accuracy of the optical flow information estimation.
[0067] In some embodiments, the cost features of the first pixel point can be retrieved from the cost memory information. For example, the cost query can be used to retrieve the cost features from the cost memory information, and the recurrent attention decoder layer can be used to iteratively optimize the prediction of the optical flow information.
[0068] After retrieving the cost features of the first pixel point, the optical flow information of the first pixel point can be predicted by combining the context features of the first pixel point.
[0069] Optionally, in some embodiments, the context features of the first pixel point in the source image can be extracted based on the third network model, where the structure of the third network model is a CNN network structure.
[0070] Thus, since the CNN network structure is a lightweight network structure and is convenient for deployment, extracting the context features of the first pixel point in the source image based on the third network model with the CNN network structure can facilitate the deployment and implementation on mobile devices, effectively expand the application scenarios, and improve the practicality.
[0071] Step S207: Align the source image and the target image according to the optical flow information of each first pixel point.
[0072] After obtaining the optical flow information of each first pixel point, the source image and the target image can also be aligned with reference to the optical flow information to form an image sequence for supporting subsequent image processing.
[0073] In this embodiment, a source image corresponding to a first image is obtained, and a target image corresponding to a second image is obtained, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after that of the first image. The similarity between each first pixel point in the source image and each second pixel point in the target image is determined to generate a cost volume of the source image, where the cost volume includes the similarity corresponding to each first pixel point, and based on the cost volume of the source image, the optical flow information of each first pixel point is determined. This is to reduce the computational complexity, save computing resources, and improve efficiency. By generating a cost map corresponding to a first pixel point according to the similarity corresponding to the first pixel point, and generating a cost volume of the source image based on a plurality of cost maps respectively corresponding to a plurality of first pixel points, where the cost volume includes the similarity corresponding to each first pixel point, it is possible to support subsequent efficient encoding processing of the cost volume of the source image, and improve the recognition efficiency and recognition effect of the optical flow information. After encoding the cost volume of the source image to obtain cost memory information, the optical flow information of each first pixel point can be determined by combining the context features of each first pixel point in the source image and the cost memory information. Thus, the estimation accuracy and efficiency of the optical flow information are improved.
[0074] The following is an example for the above embodiment:
[0075] As Figure 3 shown, Figure 3 is a schematic application diagram in an embodiment of the present disclosure. In the embodiment of the present disclosure, during the training phase, both the training data Source and Target are a pair of RAW-format original image input pairs (an optional example of a sample source image and a sample target image for training a model), and supervised training is performed using the Flow groundtruth in the training set, gradually converging. During the usage phase, generally the camera will obtain a series of burst image pairs (assumed to be n) taken in succession. The model takes the 0th frame image as the target (an optional example of the source image) and calculates the optical flow results (an optional example of the optical flow information) of flow_1, flow_2... flow_n-1 between each frame (an optional example of the target image) and the 0th frame respectively, and then uses the warp algorithm to align the 1st to n-1th frames with the original target0 frame image to obtain a series of burst-aligned image sequences for subsequent image processing. The feature extraction and similarity calculation processes both adopt a more lightweight CNN architecture for easy deployment. The channels of the cost volume encoder and the cost memory decoder can also be pruned and compressed, so that the number of parameters of the model is greatly reduced. Secondly, the input for both training and inference is changed from a three-channel RGB-format image to a single-channel RAW-format image.
[0076] Figure 4 The structural schematic diagram of an image optical flow estimation device provided by an embodiment of the present disclosure.
[0077] As Figure 4 shown, the image optical flow estimation device 40 includes:
[0078] An acquisition module 401, configured to acquire a source image corresponding to a first image and acquire a target image corresponding to a second image, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after that of the first image.
[0079] A first determination module 402, configured to determine the similarity between each first pixel point in the source image and each second pixel point in the target image.
[0080] A generation module 403, configured to generate a cost volume of the source image, where the cost volume includes the similarity corresponding to each first pixel point.
[0081] A second determination module 404, configured to determine the optical flow information of each first pixel point according to the cost volume of the source image.
[0082] It should be noted that the foregoing explanation of the embodiment of the image optical flow estimation method is also applicable to the image optical flow estimation device of this embodiment, and will not be elaborated here.
[0083] In this embodiment, by acquiring a source image corresponding to a first image and acquiring a target image corresponding to a second image, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after that of the first image, determining the similarity between each first pixel point in the source image and each second pixel point in the target image, generating a cost volume of the source image, where the cost volume includes the similarity corresponding to each first pixel point, and determining the optical flow information of each first pixel point according to the cost volume of the source image. To reduce the computational complexity, save computing resources, and improve efficiency.
[0084] Figure 5 The block diagram of an exemplary electronic device suitable for implementing the embodiments of the present disclosure is shown. Figure 5 The shown electronic device 12 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure. The electronic device can be, for example, an electronic device or a terminal.
[0085] As Figure 5As shown, the electronic device 12 is presented in the form of a general computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a memory 28, and a bus 18 that connects different system components (including the memory 28 and the processing unit 16).
[0086] The bus 18 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus structures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnection (PCI) bus.
[0087] The electronic device 12 typically includes a variety of computer system-readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.
[0088] The memory 28 may include computer system-readable media in the form of volatile memory, such as a Random Access Memory (RAM) 30 and / or a cache 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 5 not shown, commonly referred to as a "hard disk drive").
[0089] Although Figure 5Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (such as: Compact Disc Read Only Memory (hereinafter referred to as: CD-ROM), Digital Video Disc Read Only Memory (hereinafter referred to as: DVD-ROM) or other optical media) can be provided. In these cases, each drive can be connected to the bus 18 through one or more data medium interfaces. The memory 28 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.
[0090] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in the memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules 42 generally execute the functions and / or methods in the embodiments described in the present disclosure.
[0091] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a human body to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. And, the electronic device 12 can also communicate with one or more networks (such as a Local Area Network (hereinafter referred to as: LAN), a Wide Area Network (hereinafter referred to as: WAN) and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through the bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0092] The processing unit 16 executes various functional applications and data processing by running the programs stored in the memory 28, such as implementing the methods mentioned in the foregoing embodiments.
[0093] To implement the above embodiments, the present disclosure also provides a chip, including: The chip includes a processing circuit configured to execute the method provided in the foregoing embodiments.
[0094] Figure 6 FIG. is a schematic structural diagram of the chip provided by an embodiment of the present disclosure. Reference may be made to Figure 6 the schematic structural diagram of the chip 600 shown, but not limited thereto.
[0095] The chip 600 includes a processing circuit 601 configured to execute any of the above methods.
[0096] In some embodiments, the chip 600 further includes one or more interface circuits 602. Optionally, the interface circuit 602 is connected to the memory 603. The interface circuit 602 can be used to receive signals from the memory 603 or other devices, and the interface circuit 602 can be used to send signals to the memory 603 or other devices. For example, the interface circuit 602 can read the instructions stored in the memory 603 and send the instructions to the processing circuit 601.
[0097] In some embodiments, the interface circuit 602 executes at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 601 executes other steps.
[0098] In some embodiments, terms such as interface circuit, interface, transceiver pin, transceiver, etc. can be replaced with each other.
[0099] In some embodiments, the chip 600 further includes one or more memories 603 for storing instructions. Optionally, all or part of the memories 603 can be outside the chip 600.
[0100] To implement the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing a computer program, which when executed by a processor implements the method provided in the foregoing embodiments of the present disclosure.
[0101] To implement the above embodiments, the present disclosure also provides a computer program product, which when the instructions in the computer program product are executed by a processor, executes the method provided in the foregoing embodiments of the present disclosure.
[0102] The collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved in the present disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.
[0103] It should be noted that personal information from users should be collected for legal and reasonable purposes and not shared or sold outside of such legal uses. In addition, such collection / sharing should be carried out after obtaining the informed consent of the users, including but not limited to notifying the users to read the user agreement / user notice before using the function and signing an agreement / authorization including authorizing relevant user information. In addition, any necessary steps should be taken to safeguard and protect access to such personal information data and ensure that others with access to the personal information data comply with their privacy policies and procedures.
[0104] The present disclosure anticipates embodiments that can provide users with the option to block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by restricting data collection and deleting the data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of the users.
[0105] In the description of the foregoing embodiments, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0106] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.
[0107] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations, where the functions may be executed not in the order shown or discussed, including in a substantially simultaneous manner according to the functions involved or in a reverse order, which should be understood by those skilled in the art to which the embodiments of the present disclosure pertain.
[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection part with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.
[0109] It should be understood that various parts of the present disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0110] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.
[0111] In addition, each functional unit in various embodiments of the present disclosure may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0112] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.
Claims
1. An image optical flow estimation method, characterized in that, Including the following steps: Obtain a source image corresponding to a first image, and obtain a target image corresponding to a second image, where the number of channels of the source image is less than that of the first image, the number of channels of the target image is less than that of the second image, and the acquisition time of the second image is after the acquisition time of the first image; Determine the similarity between each first pixel point in the source image and each second pixel point in the target image; Generate a cost volume for the source image, where the cost volume includes the similarities corresponding to the first pixel points; Determine the optical flow information of each first pixel point according to the cost volume of the source image.
2. The method according to claim 1, wherein The image formats of the first image and the second image are RGB formats, and the image formats of the source image and the target image are RAW formats.
3. The method according to claim 1, wherein The determining the similarity between each first pixel point in the source image and each second pixel point in the target image includes: Process the source image based on a first network model to obtain a source feature map; Process the target image based on a second network model to obtain a target feature map; Obtain first feature information of the first pixel point from the source feature map, and obtain second feature information of each second pixel point from the target feature map; and Determine the similarity between the first pixel point and each second pixel point according to the first feature information and each second feature information.
4. The method according to claim 3, characterized in that The structure of the first network model is a convolutional neural network (CNN) structure, and the structure of the second network model is a CNN structure.
5. The method according to claim 1, wherein The generating the cost volume for the source image includes: Generate a cost map corresponding to the first pixel point according to the similarity corresponding to the first pixel point; Generate the cost volume of the source image according to a plurality of cost maps respectively corresponding to a plurality of the first pixel points.
6. The method according to claim 1, wherein The determining the optical flow information of each first pixel point according to the cost volume of the source image includes: Encode the cost volume of the source image to obtain cost memory information; Determine the optical flow information of each first pixel point according to the context features of each first pixel point in the source image and the cost memory information.
7. The method according to claim 6, wherein The determining the optical flow information of each first pixel point according to the context features of each first pixel point in the source image and the cost memory information includes: Retrieve the cost features of each first pixel point from the cost memory information; Predict the optical flow information of each first pixel point according to the cost features and the context features of each first pixel point.
8. The method according to claim 6, wherein The method further includes: Extract the context features of the first pixel point in the source image based on a third network model, where the structure of the third network model is a CNN structure.
9. The method according to any one of claims 1-8, characterized in that, The method further includes: Align the source image and the target image according to the optical flow information of each first pixel point.
10. An image optical flow estimation device, characterized in that, Including: An acquisition module, configured to acquire a source image corresponding to a first image and acquire a target image corresponding to a second image, wherein the number of channels of the source image is less than the number of channels of the first image, the number of channels of the target image is less than the number of channels of the second image, and the acquisition time of the second image is after the acquisition time of the first image; A first determination module, configured to determine the similarity between each first pixel point in the source image and each second pixel point in the target image; A generation module, configured to generate a cost volume of the source image, wherein the cost volume includes the similarity corresponding to each of the first pixel points; A second determination module, configured to determine the optical flow information of each of the first pixel points according to the cost volume of the source image.
11. An electronic device, characterized in that, Comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1-9.
12. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1-9.
13. A computer program product, characterized in that, Comprising a computer program, which when executed by a processor implements the method according to any one of claims 1-9.