Optical flow motion vector diagram determination method, apparatus and device, and readable storage medium

Abstract

The invention discloses an optical flow motion vector diagram determination method, apparatus and device, and a readable storage medium, and belongs to the technical field of video processing. The method is executed by a terminal and comprises the following steps: acquiring a first image and a second image in an input image; wherein the first image and the second image have different brightness; according to the transmission channel of each pixel in the first image and the second image, the first image and the second image are divided into a plurality of areas, and the area types of the areas comprise a low exposure area, a normal exposure area and a high exposure area; and determining optical flow motion vector diagrams of the first image and the second image according to the region type to which each region belongs.

Description

Technical Field

[0001] The present application belongs to the field of video processing technology, and specifically relates to a method, device, equipment and readable storage medium for determining an optical flow motion vector map. Background Art

[0002] With the development of terminal devices, more users use mobile terminal devices to take photos and prefer to watch photos and preview videos on mobile terminal devices. In real scenes, the color of objects has a very high dynamic range. Most terminal devices can support 24 bits of information to represent colors, but the pixel value of a single channel is 8 bits of data. Therefore, the dynamic range that can be covered by a single photo taken by an ordinary terminal device is extremely limited. Using too short an exposure time will make the low-brightness parts of the scene dark and produce noise, while using too long an exposure time will cause the high-brightness parts of the scene to be overexposed and lose details. In order to make the image present rich color details and brightness levels, High Dynamic Range (HDR) technology that combines alternate exposure of multiple frames of pictures has become a mainstream solution.

[0003] Among them, the optical flow estimation involved in traditional HDR video synthesis technology has low accuracy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment and readable storage medium for determining an optical flow motion vector map, which can achieve more accurate optical flow estimation.

[0005] In a first aspect, an embodiment of the present application provides a method for determining an optical flow motion vector map, which is executed by a terminal and includes:

[0006] Acquire a first image and a second image in an input image; wherein the first image and the second image have different brightness;

[0007] According to the transmission channel of each pixel in the first image and the second image, the first image and the second image are respectively divided into a plurality of regions, where the region types include a low exposure region, a normal exposure region, and a high exposure region;

[0008] According to the region type to which each of the regions belongs, an optical flow motion vector map of the first image and the second image is determined.

[0009] In a second aspect, an embodiment of the present application provides a device for determining an optical flow motion vector map, including:

[0010] A first processing module, used to obtain a first image and a second image in an input image; wherein the first image and the second image have different brightness;

[0011] A second processing module, configured to divide the first image and the second image into a plurality of regions according to a transmission channel of each pixel in the first image and the second image, wherein the region types include a low exposure region, a normal exposure region, and a high exposure region;

[0012] The third processing module is used to determine the optical flow motion vector map of the first image and the second image according to the area type to which each of the areas belongs.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0016] In a sixth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0017] In the embodiment of the present application, after acquiring the first image and the second image in the input image, the terminal can complete the region division of the first image and the second image by the transmission channel of each pixel in the first image and the second image to distinguish the low exposure area, the normal exposure area and the high exposure area, and then determine the optical flow motion vector diagram of the first image and the second image according to the area type to which each of the areas belongs. In particular, because the different exposure levels of the images are taken into consideration when determining the optical flow motion vector diagram, the accuracy of the optical flow motion vector diagram is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of the process of determining the optical flow motion vector diagram in an embodiment of the present application;

[0019] Figure 2 is one of the schematic diagrams of the input image;

[0020] Figure 3 This is the second schematic diagram of the input image;

[0021] Figure 4 It is a schematic diagram of channel processing of an image;

[0022] Figure 5 It is a schematic diagram of the segmentation of the channel map;

[0023] Figure 6 It is a schematic diagram of dividing the high exposure area of ​​image 1;

[0024] Figure 7 It is a schematic diagram of the optical flow estimation process;

[0025] Figure 8 This is a schematic diagram of generating an optical flow motion vector diagram;

[0026] Fig. 9 It is a schematic diagram of the terminal hardware structure;

[0027] Fig.10 This is a schematic diagram of the HDR-MV-LITE module structure;

[0028] Fig.11 It is a schematic diagram of the terminal interface;

[0029] Fig.12 This is a schematic diagram of the HDR video processing flow;

[0030] Fig.13 This is a schematic diagram of the offline HDR video processing process;

[0031] Fig.14 It is a schematic diagram of the module structure of the device for determining the optical flow motion vector diagram in an embodiment of the present application;

[0032] Fig.15 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0033] Fig.16 It is a schematic diagram of the structure of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] The following is a detailed description of the method for determining the optical flow motion vector map provided in the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, a method for determining an optical flow motion vector diagram in an embodiment of the present application is executed by a terminal, including:

[0038] Step 101: Acquire a first image and a second image in an input image; wherein the first image and the second image have different brightness.

[0039] Here, the input image is used to estimate the optical flow of the current shooting scene. For the input image, two images with brightness difference can be selected as the first image and the second image respectively to obtain its optical flow motion vector map.

[0040] Step 102, according to the transmission channel of each pixel in the first image and the second image, the first image and the second image are respectively divided into a plurality of regions, and the region types to which the regions belong include a low exposure region, a normal exposure region and a high exposure region.

[0041] In this step, the transmission channel of each pixel in the first image and the second image will be used to complete the area division of the first image and the second image to distinguish the low exposure area, the normal exposure area and the high exposure area, laying the foundation for the next step.

[0042] Step 103: determining an optical flow motion vector map of the first image and the second image according to the region type to which each of the regions belongs.

[0043] After the regions of the first image and the second image are divided in step 102, in this step, the optical flow motion vector diagrams of the first image and the second image can be determined for the low exposure region, the normal exposure region and the high exposure region.

[0044] Thus, according to the above steps 101-103, after acquiring the first image and the second image in the input image, the terminal can complete the region division of the first image and the second image by the transmission channel of each pixel in the first image and the second image, so as to distinguish the low exposure region, the normal exposure region and the high exposure region, and then determine the optical flow motion vector of the first image and the second image according to the region type to which each of the regions belongs. In particular, because the different exposure levels of the images are taken into consideration when determining the optical flow motion vector, the accuracy of the optical flow motion vector is improved.

[0045] It should be known that the method of the embodiment of the application is suitable for HDR synthesis applications of video frames with alternating long and short exposures, providing high-precision optical flow estimation for terminal image / video applications. For example, the input image may include multiple frames, and the exposure ratio may support alternating long and short exposure sequence images, such as Figure 2 The three frames of images (short, long, short) shown in FIG. 1 can also support long, medium and short sequence images, such as Figure 3 The five frames of images shown (increased from short to long) are shown. In the input image, the terminal can select each group (two images with different brightness) as the first image and the second image.

[0046] In this embodiment, the optical flow motion vector map is also called an optical flow vector map.

[0047] Optionally, dividing the first image and the second image into a plurality of regions respectively according to a transmission channel of each pixel in the first image and the second image includes:

[0048] Based on the transmission channel of each pixel in the first image and the second image, performing channel processing on the first image and the second image respectively to obtain a channel map of each channel;

[0049] Segmenting the channel map step by step based on the segmentation strategy until the variance of pixel values ​​of the obtained segmented parts is less than a first threshold;

[0050] Determine the region type to which each segmented part belongs based on a pixel threshold;

[0051] Based on the region type and the position information of the segmented portion, a plurality of regions of the first image and the second image are determined.

[0052] In this way, more effective region (also called block) division can be achieved through channel pixels.

[0053] Among them, the channel processing is to extract and splice the pixels of the same transmission channel according to different transmission channel types. Taking the first image as an example, if the image is arranged in RGGB channel mode, including one R channel, one B channel and two G channels, after the first image is processed by channel processing, the channel map of R channel (average of two channels), G channel and B channel can be obtained, as shown in Figure 4 As shown (different green fills are used to distinguish the two G channels).

[0054] The segmentation strategy may include a segmentation mode of segmentation at each level, such as segmentation at the first level according to N*M, segmentation at the second level according to N*N, etc. Of course, the segmentation mode of each level may also be the same. Figure 5 In the channel diagram of the G channel shown in the figure, the first segmentation (first level segmentation) is performed as 2*4 segmentation. Since the variance of the pixel value of a certain segmented part is not less than the first threshold, it is necessary to perform another segmentation (second level segmentation). The second segmentation is performed as 1*2 segmentation to obtain two segmented parts (the red box part and the blue box part). At this time, the variance of the pixel value of all segmented parts is less than the first threshold, and no further segmentation is required.

[0055] After a segmentation, whether to perform another segmentation requires judging whether the pixels in the segmented part are stable, that is, whether the variance is less than the first threshold. If the variance is less than the first threshold, it means that the pixels in the segmented part are stable. If the variance is not less than the first threshold, it means that the pixels in the segmented part are unstable.

[0056] The region type to which the segmented part belongs also includes a low exposure region, a normal exposure region and a high exposure region.

[0057] After completing the determination of the region type to which each segmented part belongs, for the target image for region division, such as the first image, the segments belonging to the same region type in each channel image of the first image are merged according to their positions in the first image to obtain multiple regions of the first image.

[0058] Optionally, the pixel threshold includes a second threshold and a third threshold, and the second threshold is greater than the third threshold;

[0059] The determining the region type to which each segmented part belongs based on the pixel threshold comprises:

[0060] If the average pixel value in the current segmented part is greater than or equal to the second threshold, the region type to which the current segmented part belongs is a high exposure region;

[0061] If the average pixel value in the current segmented portion is less than the second threshold value and greater than or equal to the third threshold value, the region type to which the current segmented portion belongs is a normal exposure region;

[0062] If the average pixel value in the current segmented portion is less than the third threshold, the region type to which the current segmented portion belongs is a low exposure region.

[0063] Here, in order to distinguish between low exposure areas, normal exposure areas and high exposure areas, the pixel thresholds including the second threshold and the third threshold are set. After completing the segmentation of the channel map, the type of area is determined based on the size of the mean pixel value in each segmented part and the second threshold and the third threshold.

[0064] As an optional implementation method, when the target image for region division is image 1 in the input RAW domain image, as Figure 6 As shown, first, image 1 is processed by channel division to obtain an R channel image (channel map of the R channel), a G channel image (channel map of the G channel) and a B channel image (channel map of the B channel). After that, the single-channel image is segmented step by step. After each segmentation, it is judged whether the variance of the pixel value of the segmented part is less than the first threshold. If the variance is less than the first threshold, it is considered that the pixel values ​​in the segmented part are stable and belong to the same region type; if the variance is not less than (greater than or equal to) the first threshold, it is considered that the pixel values ​​in the segmented part are relatively different, and there may be at least one type of sub-region in the segmented part. At this time, the size of the segmentation will be reduced (such as a reduction ratio of 1 / 2), and then the pixel value variance of the re-segmented part is calculated, and the judgment is continued until the variance is less than the first threshold and the channel map division is completed. When calculating the pixel value variance of the segmented part, the mean pixel value of the segmented part can also be calculated. In this way, after the channel map segmentation is completed, the size of the pixel value mean of the segmented part and the pixel threshold is judged to determine the low exposure area, the normal exposure area and the high exposure area. After the above process is completed, the low exposure area, normal exposure area and high exposure area can be fused and divided respectively. Taking the high exposure area as an example, the segmented part of the channel map of the R, G and B channels belonging to the high exposure area is selected and used as the division result of the high exposure area of ​​image 1.

[0065] In addition, considering that for a certain area type, the motion vector of the associated channel can be used to characterize the motion characteristics of other channels, for example, in the RAW domain image, the pixel arrangement is RGGB, where R and B are each a channel, and G is two channels to collect data. Therefore, in the Raw domain image pixel point, compared with the channel values ​​of R and B, the value on the G channel is higher. In addition, in the RAW domain image, the RGGB pixels are arranged in sequence, and the pixel arrangements under different channels have similar image edge characteristics.

[0066] Optionally, in this embodiment, determining the optical flow motion vector diagram of the first image and the second image according to the region type to which each region belongs includes:

[0067] Based on each of the region types, use the associated channel pixels to perform optical flow calculation to obtain optical flow information;

[0068] The optical flow information is fused to determine an optical flow motion vector map of the first image and the second image.

[0069] Therefore, for the first image and the second image, it is necessary to use channel pixels associated with the same region type to perform optical flow calculation (also called flow estimation) to obtain the optical flow information of this region type; after obtaining the optical flow information corresponding to each region type, they are fused to determine the optical flow motion vector map.

[0070] Optionally, the performing optical flow calculation based on each of the region types using associated channel pixels to obtain optical flow information includes:

[0071] Obtain a first channel map based on a first channel of each pixel in a first region of the first image;

[0072] Obtain a second channel map based on a second channel of each pixel in a second region of the second image;

[0073] Performing optical flow estimation based on the first channel map and the second channel map to determine optical flow information of a first region type;

[0074] The area type to which the first area and the second area belong is the first area type, the first area type is any one of the multiple area types, and the transmission channel type to which the first channel and the second channel belong is associated with the first area type.

[0075] If the first area type is a high exposure area, since the pixel values ​​on the R channel and B channel in the high exposure area are low, the transmission channel types associated with the high exposure area are R channel and B channel, then the first channel is R channel and B channel, the second channel is R channel and B channel, and the pixels on the R channel and B channel are used for optical flow estimation. If the first area type is a low exposure area, since the pixel values ​​on the G channel in the low exposure area are low, the transmission channel type associated with the low exposure area is G channel, then the first channel is G channel, the second channel is G channel, and the pixels on the G channel are used for optical flow estimation. If the first area type is a normal exposure area, and the transmission channel types associated with the normal exposure area are R channel, B channel, and G channel, then the first channel is all channels, the second channel is all channels, and the pixels on the four channels are used for optical flow estimation.

[0076] In this embodiment, the algorithms used for optical flow calculation include but are not limited to traditional solutions (Lucas-Kanade, Horn-Schunck, etc.) and AI solutions (such as FlowNet, Spynet, RAFT, etc.).

[0077] Of course, it should also be known that the optical flow calculation is performed on a single channel. Therefore, assuming that the first image is a long-exposure image L1 and the second image is a short-exposure image S1, Figure 7 As shown, after obtaining four channels of each pixel of the first image and the second image respectively and obtaining four channel maps, the optical flow estimation of the channel is performed based on the same transmission channel type, and then the optical flow information corresponding to each region type is determined based on each region type and the transmission channel type associated with the region type.

[0078] In this way, the fusion process of optical flow information is as follows: for the same area type, the channel map is spliced ​​in combination with the optical flow information to generate an optical flow motion vector map, such as Figure 8 As shown in the figure (different green fills are used to distinguish the two G channels). The fusion of optical flow information can also be understood as splicing and fusing the optical flow dense maps of the three channels into an optical flow picture, that is, an optical flow motion vector map.

[0079] After obtaining the optical flow motion vector map, an optical flow registration operation (warp registration) can be performed. For example, the long exposure image is registered using the optical flow motion vector map, and then fused with the short exposure image after registration, and finally a long and short exposure fused HDR image is generated.

[0080] It should be noted that in this embodiment, the hardware components of the terminal shown are as follows: Fig. 9 As shown, it includes (a) audio / video system, such as: camera (image sensor), audio equipment, display equipment; (b) chip processing system, such as central processing unit (CPU), neural network processor (NPU), graphics processing unit (GPU); (c) memory, such as system memory, local cache, etc. (d) communication bus. Of course, other hardware parts such as communication interface can also be included. Software components include kernel layer, system layer, application framework layer, and application layer. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor executes instructions in the memory, combined with its hardware completion.

[0081] The camera converts the optical image into an electronic signal (image Raw data), and the processor (CPU, NPU, GPU) is used for system coordination, scheduling, management, image-related processing flow and AI neural network processing flow, providing systematic computing support for the algorithm model. The system memory is used to temporarily store the calculation data in the CPU and exchange cache data with the CPU. The local cache is a buffer for CPU data exchange, which is used to solve the speed difference problem between the CPU speed and the memory speed. The processor, memory, and communication structure are connected through a communication bus; the memory is used to store execution instructions, the processor executes calculation instructions, and the required previous and next frame image data. The display device is used to present the digital image signal on the display to ensure the normal operation of the LCD display. The chip processing system is responsible for processing the data captured by the camera in the HDR video (High Dynamic Range Video Module-Lite, HDR-VM-Lite) module to output HDR-fused RGB images and HDR videos. The details of the HDR-MV-LITE module are as follows: Fig.10 As shown in the figure, it mainly includes four submodules: (a) HDR video frame acquisition submodule; (b) HDR single frame synthesis submodule; (c) storage submodule; (d) HDR video single frame processing submodule. The HDR-MV-LITE module inputs continuous long and short exposure RAW domain images and outputs RAW domain HDR synthesis images. The HDR video acquisition strategy (the strategy used by the image sensor to acquire images) is located in the HDR video frame acquisition module.

[0082] The HDR-VM-Lite module is used for terminal photo taking, video recording, video calling, live broadcasting and other functions.

[0083] Among them, the HDR single-frame synthesis submodule includes (a) optical flow estimation unit 1, (b) long and short exposure registration unit, and (c) long and short exposure synthesis unit.

[0084] The optical flow estimation unit 1 is used to estimate the optical flow motion vector between the long exposure and short exposure frames, such as applying the method for determining the optical flow motion vector map of the embodiment of the present application.

[0085] The long-short exposure registration unit is used to align the pixels of the long-short exposure images according to the optical flow motion information, and can be calculated using the following formula I(x',y')=I(x+Δx,y+Δy). Where I represents the pixel value, (x',y') represents the pixel position of the target image; (x,y) represents the pixel position of the original image, and (Δx,Δy) represents the optical flow motion vector calculated by the optical flow estimation unit 1.

[0086] The long and short exposure synthesis unit processes the long and short exposure images separately and outputs a fusion weight map. The fusion weight can be calculated based on factors such as exposure ratio, pixel brightness, regional similarity, and time similarity, and can also be output through an AI neural network.

[0087] Among them, the HDR video single frame processing submodule includes an optical flow estimation unit 2, a preprocessing unit, an HDR multi-frame feature extraction unit, an HDR multi-frame feature association unit, an HDR single-frame feature generation unit and an artifact elimination unit. The optical flow estimation unit 2 can also apply the method for determining the optical flow motion vector map of the embodiment of the present application. The optical flow estimation result is used for HDR multi-frame feature fusion. After extracting the HDR video frame features, an alignment operation is performed on the feature layer. Finally, a jitter-free, color-stable HDR video is output.

[0088] Among them, the HDR video frame acquisition submodule automatically adjusts the fusion strategy, acquisition interval, and sensitivity according to the light intensity of the external environment and the movement of objects to ensure the optimal front-end long and short frame input in different scenarios.

[0089] In one embodiment, the HDR-VM-Lite module is input as a RAW domain image, which is directly acquired by the front-end camera sensor. The pixel value is proportional to the exposure time, brightness, etc., and is linear data. Most traditional optical flow estimation is applied to RGG domain nonlinear feature images after demosaicing, and is not applicable to RAW domain linear feature images. In addition, compared with the RGB domain, the pixel arrangement distribution in the Raw domain is RGGB, so the Raw domain pixels have a higher value on the G (green) channel, and the pixel values ​​on the B (blue) and R (red) channels are relatively low. The method for determining the optical flow motion vector map of the embodiment of the present application improves the accuracy of inter-frame optical flow estimation and reduces HDR video fusion anomalies on the basis of ensuring the real-time performance of the algorithm, thereby improving the user's terminal shooting and reading experience.

[0090] Specifically, when a user uses a terminal to enable HDR video mode, such as clicking Fig.11 The HDR mode function button shown in the figure, the terminal camera will capture video frame images with a fixed frame rate and alternating long and short exposures. After that, the HDR video processing flow is as follows Fig.12 As shown:

[0091] The terminal can determine at least one fusion strategy and input the target image (a group of images) into the HDR single-frame synthesis submodule. The HDR single-frame synthesis submodule selects alternating long and short exposure frames from the DDR (a) memory for corresponding HDR single-frame fusion; at the same time, the optical flow estimation unit 1 in the HDR single-frame synthesis submodule applies the method for determining the optical flow motion vector diagram of the embodiment of the present application, and calculates the optical flow information between the input long and short exposure frames based on the first image and the second image in the target image, determines the optical flow motion vector diagram, and stores the calculation result in the DDR (c) memory; the HDR single-frame synthesis submodule saves one copy of the long and short exposure fusion result in the DDR (b) memory for subsequent HDR video single-frame processing submodule use, and one copy is directly input into the HDR video single-frame processing submodule. The HDR video single-frame processing submodule reads the original long and short exposure data, optical flow data (such as optical flow motion vector diagram), and HDR fusion picture, and finally outputs an HDR picture that is consistent with the texture, color, motion and other features of the historical frame as a frame in the HDR output video.

[0092] In the HDR video processing flow, optical flow calculation runs on the SoC GPU, and other functions of the HDR single-frame synthesis submodule are implemented on the SoC CPU; other functions of the HDR video single-frame processing submodule are implemented on the SoC NPU. The performance of each SoC chip is fully utilized to output HDR video data in real time while ensuring the stability of HDR video.

[0093] In this way, the real-time HDR video output function can be completed on the terminal side, combining the optical flow information of the HDR video frame and relying on the AI ​​neural network model to improve the front and back stability of the HDR video frame, so as to achieve real-time online HDR video output. This solution can be used to reduce the jitter phenomenon of HDR video, and HDR video truly reflects the color range of real scenes.

[0094] Or, specifically, after using the terminal for HDR shooting, the user can set the HDR offline processing function. After the setting is completed, the terminal background starts the HDR video synthesis effect when the current process task is not busy. At this time, the original long and short exposure data collected by the front-end camera has been cleared from the DDR, so it is necessary to perform frame insertion processing on the synthesized HDR video, and at the same time call the optical flow model to calculate the optical flow vector information of the front and back frames of the HDR video. The offline processing process is as follows Fig.13 As shown:

[0095] The HDR video is first deframed, and the result is stored in the DDR (a) memory; the deframed image enters the optical flow estimation unit 1 to output the optical flow motion vector information (such as the optical flow motion vector diagram) between the previous and next frames; the HDR video single frame processing submodule is called, and the HDR video single frame processing submodule performs self-training on the HDR video. After the training is completed, the trained model is used to read the HDR deframe data to regenerate each frame in the HDR video to remove the HDR video jitter. In this way, the HDR video offline output function can be completed on the terminal side, and self-training can be performed on the HDR video to make the model more compatible with the HDR video content and generate a more stable HDR video.

[0096] It should be noted that the terminal uses the CPU to send a control strategy to allocate corresponding GPU resources for the optical flow calculation of each block, and parallel calculation accelerates the optical flow calculation results of the picture to predict the optical flow motion vector field between long and short exposures. Taking the use of Lucas-Kanade for optical flow estimation as an example, before calculating the optical flow motion information of long and short exposures (such as the optical flow motion vector map), the two images need to be converted into a unified grayscale image, and then the optical flow information of the two images is calculated according to the Lucas-Kanade method. After obtaining the optical flow motion vector information, the pixel point A of the previous frame can be moved to a new position according to the motion vector, and the pixel point B of the current frame can be moved to the next frame position according to the motion vector.

[0097] The optical flow motion vector diagram determination method provided in the embodiment of the present application may be performed by an optical flow motion vector diagram determination device. The optical flow motion vector diagram determination method performed by the optical flow motion vector diagram determination device in the embodiment of the present application is taken as an example to illustrate the optical flow motion vector diagram determination device provided in the embodiment of the present application.

[0098] like Fig.14 As shown, a device 1400 for determining an optical flow motion vector diagram in an embodiment of the present application includes:

[0099] The first processing module 1410 is used to obtain a first image and a second image in an input image; wherein the first image and the second image have different brightness;

[0100] A second processing module 1420, configured to divide the first image and the second image into a plurality of regions according to a transmission channel of each pixel in the first image and the second image, wherein the region types include a low exposure region, a normal exposure region, and a high exposure region;

[0101] The third processing module 1430 is configured to determine an optical flow motion vector map of the first image and the second image according to the region type to which each region belongs.

[0102] Optionally, the second processing module is further used for:

[0103] Based on the transmission channel of each pixel in the first image and the second image, performing channel processing on the first image and the second image respectively to obtain a channel map of each channel;

[0104] Segmenting the channel map step by step based on the segmentation strategy until the variance of pixel values ​​of the obtained segmented parts is less than a first threshold;

[0105] Determine the region type to which each segmented part belongs based on a pixel threshold;

[0106] Based on the region type and the position information of the segmented portion, a plurality of regions of the first image and the second image are determined.

[0107] Optionally, the third processing module is further used for:

[0108] Obtaining optical flow information of each of the region types;

[0109] The optical flow information is fused to determine an optical flow motion vector map of the first image and the second image.

[0110] After acquiring the first image and the second image in the input image, the device can complete the region division of the first image and the second image by the transmission channel of each pixel in the first image and the second image to distinguish the low exposure region, the normal exposure region and the high exposure region, and then determine the optical flow motion vector diagram of the first image and the second image according to the region type to which each of the regions belongs. In particular, because the different exposure levels of the images are taken into consideration when determining the optical flow motion vector diagram, the accuracy of the optical flow motion vector diagram is improved.

[0111] The device for determining the optical flow motion vector map in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0112] The optical flow motion vector diagram determining device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0113] The optical flow motion vector diagram determination device provided in the embodiment of the present application can achieve Figure 1-Figure 13 To avoid repetition, the various processes implemented in the embodiment of the method for determining the optical flow motion vector map are not described here.

[0114] Alternatively, if Fig.15 As shown, an embodiment of the present application also provides an electronic device 1500, including a processor 1501 and a memory 1502, and the memory 1502 stores a program or instruction that can be executed on the processor 1501. When the program or instruction is executed by the processor 1501, the various steps of the embodiment of the method for determining the optical flow motion vector map mentioned above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0115] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0116] Fig.16 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0117] The electronic device 1600 includes but is not limited to: a radio frequency unit 1601, a network module 1602, an audio output unit 1603, an input unit 1604, a sensor 1605, a display unit 1606, a user input unit 1607, an interface unit 1608, a memory 1609, and a processor 1610 and other components.

[0118] Those skilled in the art will appreciate that the electronic device 1600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.16 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0119] The processor 1610 is configured to obtain a first image and a second image in an input image; wherein the first image and the second image have different brightness;

[0120] According to the transmission channel of each pixel in the first image and the second image, the first image and the second image are respectively divided into a plurality of regions, where the region types include a low exposure region, a normal exposure region, and a high exposure region;

[0121] According to the region type to which each of the regions belongs, an optical flow motion vector map of the first image and the second image is determined.

[0122] After acquiring the first image and the second image in the input image, the electronic device can complete the region division of the first image and the second image by the transmission channel of each pixel in the first image and the second image to distinguish the low exposure region, the normal exposure region and the high exposure region, and then determine the optical flow motion vector map of the first image and the second image according to the region type to which each of the regions belongs. In particular, because the different exposure levels of the images are taken into consideration when determining the optical flow motion vector map, the accuracy of the optical flow motion vector map is improved.

[0123] Optionally, the processor 1610 is further configured to:

[0124] Based on the transmission channel of each pixel in the first image and the second image, performing channel processing on the first image and the second image respectively to obtain a channel map of each channel;

[0125] Segmenting the channel map step by step based on the segmentation strategy until the variance of pixel values ​​of the obtained segmented parts is less than a first threshold;

[0126] Determine the region type to which each segmented part belongs based on a pixel threshold;

[0127] Based on the region type and the position information of the segmented portion, a plurality of regions of the first image and the second image are determined.

[0128] Optionally, the pixel threshold includes a second threshold and a third threshold, and the second threshold is greater than the third threshold;

[0129] The processor 1610 is further configured to:

[0130] If the average pixel value in the current segmented part is greater than or equal to the second threshold, the region type to which the current segmented part belongs is a high exposure region;

[0131] If the average pixel value in the current segmented portion is less than the second threshold value and greater than or equal to the third threshold value, the region type to which the current segmented portion belongs is a normal exposure region;

[0132] If the average pixel value in the current segmented portion is less than the third threshold, the region type to which the current segmented portion belongs is a low exposure region.

[0133] Optionally, the processor 1610 is further configured to:

[0134] Based on each of the region types, use the associated channel pixels to perform optical flow calculation to obtain optical flow information;

[0135] The optical flow information is fused to determine an optical flow motion vector map of the first image and the second image.

[0136] Optionally, the processor 1610 is further configured to:

[0137] Obtain a first channel map based on a first channel of each pixel in a first region of the first image;

[0138] Obtain a second channel map based on a second channel of each pixel in a second region of the second image;

[0139] Performing optical flow estimation based on the first channel map and the second channel map to determine optical flow information of a first region type;

[0140] The area type to which the first area and the second area belong is the first area type, the first area type is any one of the multiple area types, and the transmission channel type to which the first channel and the second channel belong is associated with the first area type.

[0141] It should be understood that in the embodiment of the present application, the input unit 1604 may include a graphics processor (Graphics Processing Unit, GPU) 16041 and a microphone 16042, and the graphics processor 16041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1606 may include a display panel 16061, and the display panel 16061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1607 includes a touch panel 16071 and at least one of other input devices 16072. The touch panel 16071 is also called a touch screen. The touch panel 16071 may include two parts: a touch detection device and a touch controller. Other input devices 16072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0142] The memory 1609 can be used to store software programs and various data. The memory 1609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1609 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0143] The processor 1610 may include one or more processing units; optionally, the processor 1610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1610.

[0144] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned method for determining the optical flow motion vector diagram is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0145] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0146] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the method for determining the optical flow motion vector diagram, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0147] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0148] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method for determining the optical flow motion vector diagram, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0149] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0151] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for determining an optical flow motion vector map, characterized in that: Executed by the terminal, including: Acquire a first image and a second image in an input image; wherein the first image and the second image have different brightness; According to the transmission channel of each pixel in the first image and the second image, the first image and the second image are respectively divided into a plurality of regions, where the region types include a low exposure region, a normal exposure region, and a high exposure region; According to the region type to which each of the regions belongs, an optical flow motion vector map of the first image and the second image is determined.

2. The method according to claim 1, characterized in that The step of dividing the first image and the second image into a plurality of regions according to the transmission channel of each pixel in the first image and the second image comprises: Based on the transmission channel of each pixel in the first image and the second image, performing channel processing on the first image and the second image respectively to obtain a channel map of each channel; Segmenting the channel map step by step based on the segmentation strategy until the variance of pixel values ​​of the obtained segmented parts is less than a first threshold; Determine the region type to which each segmented part belongs based on a pixel threshold; Based on the region type and the position information of the segmented portion, a plurality of regions of the first image and the second image are determined.

3. The method according to claim 2, characterized in that The pixel threshold includes a second threshold and a third threshold, and the second threshold is greater than the third threshold; The determining the region type to which each segmented part belongs based on the pixel threshold comprises: If the average pixel value in the current segmented part is greater than or equal to the second threshold, the region type to which the current segmented part belongs is a high exposure region; If the average pixel value in the current segmented portion is less than the second threshold value and greater than or equal to the third threshold value, the region type to which the current segmented portion belongs is a normal exposure region; If the average pixel value in the current segmented portion is less than the third threshold, the region type to which the current segmented portion belongs is a low exposure region.

4. The method according to claim 1, characterized in that: The step of determining the optical flow motion vector diagram of the first image and the second image according to the region type to which each region belongs includes: Based on each of the region types, use the associated channel pixels to perform optical flow calculation to obtain optical flow information; The optical flow information is fused to determine an optical flow motion vector map of the first image and the second image.

5. The method according to claim 4, characterized in that The step of performing optical flow calculation based on each of the region types using associated channel pixels to obtain optical flow information includes: Obtain a first channel map based on a first channel of each pixel in a first region of the first image; Obtain a second channel map based on a second channel of each pixel in a second region of the second image; Performing optical flow estimation based on the first channel map and the second channel map to determine optical flow information of a first region type; The area type to which the first area and the second area belong is the first area type, the first area type is any one of the multiple area types, and the transmission channel type to which the first channel and the second channel belong is associated with the first area type.

6. A device for determining an optical flow motion vector map, characterized in that: include: A first processing module, used to obtain a first image and a second image in an input image; wherein the first image and the second image have different brightness; A second processing module, configured to divide the first image and the second image into a plurality of regions according to a transmission channel of each pixel in the first image and the second image, wherein the region types include a low exposure region, a normal exposure region, and a high exposure region; The third processing module is used to determine the optical flow motion vector map of the first image and the second image according to the area type to which each of the areas belongs.

7. The device according to claim 6, characterized in that The second processing module is also used for: Based on the transmission channel of each pixel in the first image and the second image, performing channel processing on the first image and the second image respectively to obtain a channel map of each channel; Segmenting the channel map step by step based on the segmentation strategy until the variance of pixel values ​​of the obtained segmented parts is less than a first threshold; Determine the region type to which each segmented part belongs based on a pixel threshold; Based on the region type and the position information of the segmented portion, a plurality of regions of the first image and the second image are determined.

8. The device according to claim 6, characterized in that The third processing module is also used for: Obtaining optical flow information of each of the region types; The optical flow information is fused to determine an optical flow motion vector map of the first image and the second image.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for determining the optical flow motion vector map as described in any one of claims 1 to 5 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for determining the optical flow motion vector map according to any one of claims 1 to 5 are implemented.

Images