Abnormal optical flow detection method and device

Abstract

The invention relates to an abnormal optical flow detection method and device. The method comprises the following steps: determining optical flow information between a first image and a second image; super-pixel information of the reference image is determined, wherein the super-pixel information is a super-pixel segmentation result obtained after super-pixel segmentation is carried out on the reference image; the reference image is a first image or a second image; determining image gradient information and optical flow gradient information of a superpixel edge according to the superpixel information and the optical flow information; and detecting whether an abnormal optical flow exists in the optical flow information or not according to the image gradient information and the optical flow gradient information of the superpixel edge. By implementing the embodiment of the invention, the operand can be greatly saved; the method is suitable for optical flow detection in difficult scenes such as large-scale motion, weak texture, repeated texture and target shielding, and the detection effect of wrong optical flow can be improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of digital image processing and video processing, and in particular, to an abnormal optical flow detection method and apparatus thereof. Background Art

[0002] Optical flow refers to the instantaneous velocity of the pixels of a spatially moving object on the observed imaging plane, and is generally used to describe the motion relationship of corresponding positions in two frames of images. Optical flow has important applications in image registration, motion detection, target tracking, etc. In actual use scenarios, due to various difficult scenarios such as large-scale motion, weak texture, repetitive texture, and target occlusion, the optical flow calculation may fail in some areas, affecting the effect of subsequent image processing algorithms. Therefore, after the optical flow calculation is completed, the optical flow results are usually evaluated and screened to achieve the purpose of abnormal optical flow detection.

[0003] However, there is currently a lack of effective means for detecting abnormal optical flow. Summary of the Invention

[0004] The present disclosure provides an abnormal optical flow detection method and apparatus thereof.

[0005] In a first aspect, an embodiment of the present disclosure provides an abnormal optical flow detection method, including:

[0006] Determining the optical flow information between a first image and a second image;

[0007] Determining the superpixel information of a reference image, where the superpixel information is the superpixel segmentation result obtained by performing superpixel segmentation on the reference image; the reference image is the first image or the second image;

[0008] Determining the image gradient information and optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information;

[0009] Detecting whether there is abnormal optical flow in the optical flow information according to the image gradient information and optical flow gradient information of the superpixel edge.

[0010] In the above embodiment, by introducing the semantic information of superpixels, the optical flow quality is evaluated at the scale of superpixels, and the screening of abnormal optical flow is completed. Since the present disclosure only focuses on the image and optical flow information at the superpixel segmentation edge, there is no need to traverse all pixels, which can greatly save the amount of computation; in addition, superpixels can provide spatial semantic information, which is robust to repetitive texture, weak texture, and occlusion scenarios, and can be applied to optical flow detection in difficult scenarios such as large-scale motion, weak texture, repetitive texture, and target occlusion, and the screening of abnormal optical flow is more direct, improving the detection effect of incorrect optical flow.

[0011] In some embodiments in combination with some embodiments of the first aspect, determining the image gradient information and the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information includes: determining the image gradient information of the superpixel edge according to the superpixel information and the pixel information of the reference image; and determining the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information.

[0012] In the above embodiments, the gradient calculations are respectively performed on the image superpixels and the optical flow superpixel edges to obtain the gradient information of both, which facilitates subsequent screening of abnormal optical flow according to the gradient information, thereby completing the detection of abnormal optical flow. There is no need to traverse all the pixels of the entire image, which can greatly save the amount of computation.

[0013] In some embodiments in combination with some embodiments of the first aspect, determining the image gradient information of the superpixel edge according to the superpixel information and the pixel information of the reference image includes: determining the pixel values of the pixels around the superpixel edge in the reference image according to the superpixel information; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; and statistically calculating the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions according to the pixel values of the pixels around the superpixel edge to obtain the image gradient information of the superpixel edge.

[0014] In some embodiments in combination with some embodiments of the first aspect, determining the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information includes: determining the optical flow values of the pixels around the superpixel edge in the optical flow information according to the superpixel information; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; and statistically calculating the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions according to the optical flow values of the pixels around the superpixel edge to obtain the optical flow gradient information of the superpixel edge.

[0015] In some embodiments in combination with some embodiments of the first aspect, the method further includes: determining the image gradient value of the non-superpixel edge region in the reference image as a first value; and / or determining the optical flow gradient value of the non-superpixel edge region in the optical flow information as the first value.

[0016] In combination with some embodiments of the first aspect, in some embodiments, detecting whether there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge includes at least one of the following: determining that there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge; determining the type of the abnormal optical flow according to the image gradient information and the optical flow gradient information of the superpixel edge.

[0017] In combination with some embodiments of the first aspect, in some embodiments, determining that there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge includes at least one of the following: when the image gradient information of the superpixel edge is less than or equal to a first image gradient threshold and the optical flow gradient information of the superpixel edge is greater than or equal to a first optical flow gradient threshold, determining that the optical flow associated with the superpixel edge is abnormal optical flow; when the image gradient information of the superpixel edge is greater than or equal to the first image gradient threshold, and / or, the optical flow gradient information of the superpixel edge is less than or equal to the first optical flow gradient threshold, determining the image gradient information of the superpixel edge at the boundary of the target region, the image gradient information of the superpixel edge inside the target region, the optical flow gradient information of the superpixel edge at the boundary of the target region, and the optical flow gradient information of the superpixel edge inside the target region; the target region is a target region identified from the reference image by using a target detection algorithm; when the image gradient information of the superpixel edge at the boundary of the target region is greater than or equal to a second image gradient threshold, the image gradient information of the superpixel edge inside the target region is less than or equal to the first image gradient threshold, and the optical flow gradient information of the superpixel edge at the boundary of the target region and the optical flow gradient information of the superpixel edge inside the target region are both greater than or equal to the first optical flow gradient threshold, determining that the optical flow associated with the target region is abnormal optical flow; wherein, the second image gradient threshold is greater than the first image gradient threshold.

[0018] In a second aspect, an embodiment of the present disclosure provides an abnormal optical flow detection device, including:

[0019] A first determination module, configured to determine optical flow information between a first image and a second image;

[0020] A second determination module, configured to determine superpixel information of a reference image, where the superpixel information is a superpixel segmentation result obtained by performing superpixel segmentation on the reference image; the reference image is the first image or the second image;

[0021] A third determination module, configured to determine image gradient information and optical flow gradient information of a superpixel edge according to the superpixel information and the optical flow information;

[0022] A detection module, configured to detect whether there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge.

[0023] Combined with some embodiments of the second aspect, in some embodiments, the third determination module includes: a first determination unit, configured to determine the image gradient information of the superpixel edge according to the superpixel information and the pixel information of the reference image; a second determination unit, configured to determine the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information.

[0024] Combined with some embodiments of the second aspect, in some embodiments, the first determination unit is specifically configured to: determine the pixel values of the pixels around the superpixel edge in the reference image according to the superpixel information; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; according to the pixel values of the pixels around the superpixel edge, statistically calculate the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions to obtain the image gradient information of the superpixel edge.

[0025] Combined with some embodiments of the second aspect, in some embodiments, the second determination unit is specifically configured to: determine the optical flow values of the pixels around the superpixel edge in the optical flow information according to the superpixel information; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; according to the optical flow values of the pixels around the superpixel edge, statistically calculate the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions to obtain the optical flow gradient information of the superpixel edge.

[0026] Combined with some embodiments of the second aspect, in some embodiments, the apparatus further includes: a fourth determination module configured to: determine the image gradient value of the non-superpixel edge area in the reference image as a first value; and / or, determine the optical flow gradient value of the non-superpixel edge area in the optical flow information as the first value.

[0027] Combined with some embodiments of the second aspect, in some embodiments, the detection module is configured to perform at least one of the following: determine that there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge; determine the type of the abnormal optical flow according to the image gradient information and the optical flow gradient information of the superpixel edge.

[0028] In combination with some embodiments of the second aspect, in some embodiments, the detection module is configured to perform at least one of the following: determining that the optical flow associated with the superpixel edge is abnormal optical flow when the image gradient information of the superpixel edge is less than or equal to a first image gradient threshold and the optical flow gradient information of the superpixel edge is greater than or equal to a first optical flow gradient threshold; determining the image gradient information of the superpixel edge at the boundary of the target region, the image gradient information of the superpixel edge inside the target region, the optical flow gradient information of the superpixel edge at the boundary of the target region, and the optical flow gradient information of the superpixel edge inside the target region when the image gradient information of the superpixel edge is greater than or equal to the first image gradient threshold, and / or, the optical flow gradient information of the superpixel edge is less than or equal to the first optical flow gradient threshold; the target region is a target region identified from the reference image by using a target detection algorithm; determining that the optical flow associated with the target region is abnormal optical flow when the image gradient information of the superpixel edge at the boundary of the target region is greater than or equal to a second image gradient threshold, the image gradient information of the superpixel edge inside the target region is less than or equal to the first image gradient threshold, and the optical flow gradient information of the superpixel edge at the boundary of the target region and the optical flow gradient information of the superpixel edge inside the target region are both greater than or equal to the first optical flow gradient threshold; wherein, the second image gradient threshold is greater than the first image gradient threshold.

[0029] In a third aspect, an embodiment of the present disclosure provides a communication device, including: one or more processors; wherein, the processor is configured to call instructions to cause the communication device to execute the abnormal optical flow detection method described in the foregoing first aspect.

[0030] In a fourth aspect, an embodiment of the present disclosure provides a storage medium, and the foregoing storage medium stores instructions, when the foregoing instructions run on a communication device, causing the communication device to execute the method described in the optional implementation manner of the foregoing first aspect.

[0031] In a fifth aspect, an embodiment of the present disclosure provides a program product, when the foregoing program product is executed by a communication device, causing the communication device to execute the method described in the optional implementation manners of the x-th aspect and the x-th aspect, the x-th aspect and the x-th aspect.

[0032] In a sixth aspect, an embodiment of the present disclosure provides a computer program, when it runs on a computer, causing the computer to execute the method described in the optional implementation manner of the first aspect.

[0033] In a seventh aspect, an embodiment of the present disclosure provides a chip or a chip system. The chip or the chip system includes a processing circuit configured to execute the method described in the optional implementation manner of the foregoing first aspect.

[0034] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present disclosure. Description of the Drawings

[0035] The drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.

[0036] Figure 1 is a flowchart of an abnormal optical flow detection method shown according to an exemplary embodiment.

[0037] Figure 2 is a flowchart of an abnormal optical flow detection method shown according to an exemplary embodiment.

[0038] Figure 3 is a schematic diagram of the process of an abnormal optical flow detection method shown according to an exemplary embodiment.

[0039] Figure 4a is an example diagram of a superpixel segmentation result shown according to an exemplary embodiment.

[0040] Figure 4b For Figure 4a is an example diagram of the visualization result of the superpixel edge gradient after calculating the gradient information of the image shown.

[0041] Figure 5a is an example diagram of an optical flow map shown according to an exemplary embodiment.

[0042] Figure 5b For Figure 5a is an example diagram of the visualization result of the superpixel edge gradient of the optical flow map after calculating the gradient information of the optical flow information shown.

[0043] Figure 6a is an example diagram of a superpixel segmentation result shown according to an exemplary embodiment.

[0044] Figure 6b For Figure 6a is an example diagram of the corresponding optical flow result.

[0045] Figure 7a is an example diagram of a superpixel segmentation result of an occluded or moving area shown according to an exemplary embodiment.

[0046] Figure 7b For Figure 7a is a schematic diagram of the optical flow map corresponding to the image shown.

[0047] Figure 7c For Figure 7a is an example diagram of the visualization result of the superpixel edge gradient after calculating the gradient information of the image shown.

[0048] Figure 7d For Figure 7b Example diagram of the visualization result of the superpixel edge gradient after calculating the gradient information for the shown optical flow diagram.

[0049] Figure 8 Schematic flowchart of abnormal optical flow detection shown according to an exemplary embodiment.

[0050] Figure 9 Block diagram of an abnormal optical flow detection device shown according to an exemplary embodiment.

[0051] Figure 10 Schematic diagram of the structure of the communication device 1000 proposed in the embodiments of the present disclosure.

[0052] Figure 11 Schematic diagram of the structure of the chip 1100 proposed in the embodiments of the present disclosure. Detailed implementation manners

[0053] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0054] It should be noted that in the description of the present disclosure, unless otherwise stated, " / " means "or". For example, A / B may represent A or B; the "and / or" herein is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0055] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present disclosure. The singular forms "a" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0056] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "when" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0057] It should be noted that in the technical solutions of the present disclosure, the processing of the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information complies with the provisions of relevant laws and regulations and does not violate public order and good customs.

[0058] The embodiments of the present disclosure are not exhaustive, but only schematic illustrations of some embodiments, and do not constitute specific limitations on the protection scope of the present disclosure. Without contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily. For example, the solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation manners in a certain embodiment can be combined arbitrarily; furthermore, the embodiments can be combined arbitrarily. For example, some or all of the steps of different embodiments can be combined arbitrarily, and a certain embodiment can be combined arbitrarily with the optional implementation manners of other embodiments.

[0059] In each embodiment of the present disclosure, if there is no special description and logical conflict, the terms and / or descriptions between the embodiments are consistent and can be cited from each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0060] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure.

[0061] In some embodiments, devices, etc. can be interpreted as physical or virtual, and their names are not limited to those recorded in the embodiments. Terms such as "device", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject", etc. can be replaced with each other.

[0062] Optical flow refers to the instantaneous velocity of the pixel motion of a moving object in space on the observed imaging plane, and is generally used to describe the motion relationship of corresponding positions in two frames of images. Generally, optical flow can be divided into sparse optical flow and dense optical flow, both of which have important applications in image registration, motion detection, target tracking, etc. In actual usage scenarios, due to various difficult scenarios such as large-scale motion, weak texture, repetitive texture, and target occlusion, the optical flow calculation will fail in some areas, affecting the effect of subsequent image processing algorithms. Therefore, after the optical flow calculation is completed, an abnormal optical flow detection method is usually used to evaluate and screen the optical flow results, so as to avoid subsequent problems caused by incorrect optical flow.

[0063] In the related art, there are mainly the following two abnormal optical flow detection methods: One is to evaluate and screen the optical flow quality during the optical flow calculation process, such as the left-right consistency check scheme; the other is to evaluate the optical flow after the calculation is completed. Such schemes generally need to use the original image as a reference, such as by comparing the optical flow warp with the original image. However, the above two abnormal optical flow detection schemes have problems such as large computational overhead and lack of semantic information, and it is difficult to provide real-time and accurate detection results on mobile devices. In addition, for the second scheme above, it is necessary to perform a full-image warp on the image, which is computationally complex. At the same time, for regions with repetitive textures, it is impossible to determine whether the optical flow is abnormal by warping the image, and it is difficult to screen abnormal optical flow in such scenarios.

[0064] For this reason, the embodiments of the present disclosure propose an abnormal optical flow detection method based on superpixel edges. By introducing the semantic information of superpixels, the optical flow quality is evaluated at the scale of superpixels to complete the screening of abnormal optical flow. Since the present disclosure only focuses on the image and optical flow information at the superpixel segmentation edges and does not need to traverse all image pixels, the computational amount can be greatly saved. At the same time, superpixels provide semantic information in space and are robust to repetitive textures, weak textures, and occlusion scenarios.

[0065] Figure 1 is a flowchart of an abnormal optical flow detection method shown according to an exemplary embodiment. As Figure 1 shown, the abnormal optical flow detection method is used in a communication device and may include but is not limited to the following steps.

[0066] In step 101, determine the optical flow information between the first image and the second image.

[0067] In some embodiments, the first image and the second image are two consecutive frames of images in the same scene. Exemplarily, the first image and the second image may be two consecutive frames of images in a video. Exemplarily, taking the shooting scene as an example, the first image and the second image may also be two images taken continuously.

[0068] In some embodiments, the optical flow algorithm may be used to calculate the optical flow information between the first image and the second image, so as to obtain the optical flow information (also called the optical flow map) between the first image and the second image. In a possible implementation manner, the optical flow algorithm may include but is not limited to optical flow algorithms such as DIS, and may also be other optical flow algorithms, such as the Lucas-Kanade algorithm, or the Brox optical flow algorithm, etc. Here, the present disclosure does not make any limitations on this and will not elaborate.

[0069] In some embodiments, the optical flow information may be the optical flow map between the first image and the second image.

[0070] In some embodiments, the above-mentioned first image and second image may be grayscale images, or images in other color spaces. For example, they may be images of various types such as RGB, YUV, LAB (also known as CiELAB or CIE Lab), etc. The present disclosure does not limit this here and will not elaborate further.

[0071] In step 102, determine the superpixel information of the reference image.

[0072] In some embodiments, a superpixel is a small block obtained by dividing an image and is between a pixel and an image object. A superpixel is a small region composed of a series of adjacent pixels with similar features such as color, brightness, and texture. Most of these small regions retain the effective information for further image segmentation and generally do not destroy the boundary information of the objects in the image.

[0073] In some embodiments, the above-mentioned superpixel information may be the superpixel segmentation result obtained by performing superpixel segmentation on the reference image. Exemplarily, a superpixel segmentation algorithm may be used to perform superpixel segmentation on the reference image to obtain the superpixel information of the reference image. In one possible implementation, the superpixel segmentation algorithm may be the SLIC superpixel segmentation algorithm, or it may also be other superpixel segmentation algorithms, such as the SEEDS or LSC superpixel segmentation algorithms, etc. The present disclosure does not limit this here and will not elaborate further.

[0074] In some embodiments, the above-mentioned reference image may be the above-mentioned first image or second image. Exemplarily, the above-mentioned first image may be used as the reference image, or the above-mentioned second image may be used as the reference image.

[0075] In step 103, according to the superpixel information and the optical flow information, determine the image gradient information and the optical flow gradient information of the superpixel edge.

[0076] In some embodiments, after obtaining the superpixel information and the optical flow information, the gradient calculation can be performed on the image superpixel and the optical flow superpixel edge respectively according to the superpixel information and the optical flow information, so as to obtain the image gradient information of the superpixel edge and the optical flow gradient information of the superpixel edge.

[0077] Exemplarily, for the position at the superpixel edge, the pixel values of the neighboring pixels at this position can be used for gradient calculation to obtain the image gradient information at this position; or the optical flow values of the neighboring pixels at this position can be used for gradient calculation to obtain the optical flow gradient information at this position.

[0078] In step 104, according to the image gradient information and the optical flow gradient information of the superpixel edge, detect whether there is abnormal optical flow in the optical flow information.

[0079] In some embodiments, the abnormal optical flow in the optical flow information can be screened according to the image gradient information and the optical flow gradient information of the superpixel edge, so as to complete the detection of abnormal optical flow. Exemplarily, the image gradient information and the optical flow gradient information of the superpixel edge can be respectively compared with corresponding thresholds in terms of magnitude, and whether the optical flow covered by the superpixel edge is abnormal optical flow can be detected according to the comparison results. For example, if the image gradient information and the optical flow gradient information of the superpixel edge respectively meet the corresponding threshold conditions, it can be determined that the optical flow covered by the superpixel edge is abnormal optical flow. If the image gradient information and the optical flow gradient information of the superpixel edge do not meet the corresponding threshold conditions, it can be determined that the optical flow covered by the superpixel edge is not abnormal optical flow.

[0080] In the above embodiments, by introducing the semantic information of the superpixel, the quality of the optical flow is evaluated at the scale of the superpixel, and the screening of abnormal optical flow is completed. Since the present disclosure only focuses on the image and optical flow information of the superpixel segmentation edge and does not need to traverse all the pixels of the image, the computational amount can be greatly saved; in addition, the superpixel can provide semantic information in space, is robust to repeated textures, weak textures and occlusion scenes, and can be applied to optical flow detection in difficult scenes such as large-scale motion, weak texture, repeated texture, and target occlusion. The screening of abnormal optical flow is more direct, improving the detection effect of incorrect optical flow.

[0081] Figure 2 is a flowchart of an abnormal optical flow detection method shown according to an exemplary embodiment, as Figure 2 shown, the abnormal optical flow detection method is used in a communication device and may include but is not limited to the following steps.

[0082] In step 201, determine the optical flow information between the first image and the second image.

[0083] In the embodiments of the present disclosure, the implementation manner of step 201 can be implemented by any one of the embodiments of the present disclosure respectively, and no limitation is made here and will not be elaborated further.

[0084] In step 202, determine the superpixel information of the reference image, where the superpixel information is the superpixel segmentation result obtained after performing superpixel segmentation on the reference image; the reference image is the first image or the second image.

[0085] In the embodiments of the present disclosure, the implementation manner of step 202 can be implemented by any one of the embodiments of the present disclosure respectively, and no limitation is made here and will not be elaborated further.

[0086] In step 203, according to the superpixel information and the pixel information of the reference image, determine the image gradient information of the superpixel edge.

[0087] In some embodiments, after obtaining the superpixel information and the optical flow information, the gradient of the image superpixels can be calculated based on the superpixel information and the pixel information of the reference image, so that the image gradient information of the superpixel edges can be obtained.

[0088] In some embodiments, based on the superpixel information, the pixel values of the pixels around the superpixel edge in the reference image can be determined, where the pixels around the superpixel edge can include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; according to the pixel values of the pixels around the superpixel edge, the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions is statistically calculated to obtain the image gradient information of the superpixel edge.

[0089] Exemplarily, taking the position of a certain superpixel edge in the reference image as an example, the pixel change gradient of the reference image at this position is calculated. Assuming that the position coordinates are (x, y) and the pixel value is g(x, y), then the sum of the gradients in its horizontal, vertical, and diagonal directions is statistically calculated, and its gradient calculation formula is as follows:

[0090] grad_g(x,y)=abs(g(x+1,y)-g(x-1,y))+abs(g(x,y+1)-g(x,y-1))

[0091] +abs(g(x+1,y+1)-g(x-1,y-1))

[0092] +abs(g(x-1,y+1)-g(x+1,y-1))

[0093] Among them, grad_g(x, y) is the image gradient information (such as the image gradient value) of the position coordinates (x, y); abs is the gradient function used to calculate the gradient of the image; g(x + 1, y) is the pixel value of the coordinate (x + 1, y), g(-1, y) is the pixel value of the coordinate (x - 1, y), g(x, y + 1) is the pixel value of the coordinate (x, y + 1), g(x, y - 1) is the pixel value of the coordinate (x, y - 1), g(x + 1, y + 1) is the pixel value of the coordinate (x + 1, y + 1), g(x - 1, y - 1) is the pixel value of the coordinate (x - 1, y - 1), g(x - 1, y + 1) is the pixel value of the coordinate (x - 1, y + 1), and g(x + 1, y - 1) is the pixel value of the coordinate (x + 1, y - 1).

[0094] In some embodiments, the image gradient value of the non-superpixel edge region in the reference image can be determined as a first value. Exemplarily, the non-superpixel edge region (such as the region inside the superpixel) usually consists of pixel points with similar features such as color, brightness, and texture. The image gradient value of the non-superpixel edge region in the reference image can be set to the first value. For example, this first value can be 0, that is to say, the image gradient value of the non-superpixel edge region in the reference image can be directly set to 0.

[0095] In step 204, according to the superpixel information and the optical flow information, the optical flow gradient information of the superpixel edge is determined.

[0096] In some embodiments, after obtaining the superpixel information and the optical flow information, the gradient of the optical flow superpixel edge can be calculated according to the superpixel information and the optical flow information, so that the optical flow gradient information of the superpixel edge can be obtained.

[0097] In some embodiments, according to the superpixel information, the optical flow values of the pixels around the superpixel edge in the optical flow information are determined, where the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; according to the optical flow values of the pixels around the superpixel edge, the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions is statistically calculated to obtain the optical flow gradient information of the superpixel edge.

[0098] Exemplarily, taking a certain superpixel edge position in the optical flow information (also called the optical flow map) as an example, the optical flow change gradient at this position can be calculated. Assuming that the position coordinates are (x, y) and the optical flow value is mv(x, y), then the sum of the gradients in its horizontal, vertical, and diagonal directions is statistically calculated. The gradient calculation formula is as follows:

[0099] grad_mv(x,y)=abs(mv(x+1,y)-mv(x-1,y))+abs(mv(x,y+1)-mv(x,y-1))

[0100] +abs(mv(x+1,y+1)-mv(x-1,y-1))

[0101] +abs(mv(x-1,y+1)-mv(x+1,y-1))

[0102] Among them, grad_mv(x,y) is the optical flow gradient information (such as the optical flow gradient value) of the position coordinates (x,y); abs is the gradient function used to calculate the gradient of the image; mv(x+1,y) is the optical flow value of the coordinate (x+1,y), mv(-1,y) is the optical flow value of the coordinate (x-1,y), mv(x,y+1) is the optical flow value of the coordinate (x,y+1), mv(x,y-1) is the optical flow value of the coordinate (x,y-1), mv(x+1,y+1) is the optical flow value of the coordinate (x+1,y+1), mv(x-1,y-1) is the optical flow value of the coordinate (x-1,y-1), mv(x-1,y+1) is the optical flow value of the coordinate (x-1,y+1), and mv(x+1,y-1) is the optical flow value of the coordinate (x+1,y-1).

[0103] In some embodiments, the optical flow gradient value in the non-superpixel edge region among the optical flow information can be determined as the first value. Exemplarily, the non-superpixel edge region (such as the region inside the superpixel) usually consists of pixel points with similar features such as color, brightness, and texture. The optical flow gradient value in the non-superpixel edge region of the optical flow information can be set to the first value. For example, the first value can be 0, that is to say, the optical flow gradient value in the non-superpixel edge region of the optical flow information can be directly set to 0.

[0104] In step 205, according to the image gradient information and the optical flow gradient information of the superpixel edge, it is detected whether there is abnormal optical flow in the optical flow information.

[0105] In some embodiments, it can be determined that there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge. Exemplarily, the image gradient information and the optical flow gradient information of the superpixel edge can be respectively compared with the corresponding thresholds in terms of size, and whether the optical flow covered by the superpixel edge is abnormal optical flow can be detected according to the comparison results.

[0106] In some embodiments, the type of abnormal optical flow in the optical flow information can be determined based on the image gradient information and the optical flow gradient information of the superpixel edge. Exemplarily, different types of abnormal optical flow correspond to different threshold judgment conditions. It can be determined which threshold judgment condition is satisfied by the image gradient information and the optical flow gradient information of the superpixel edge, so as to determine that the optical flow covered by the superpixel edge belongs to the abnormal optical flow of the corresponding type. For example, the abnormal optical flow includes abnormal optical flow in a flat area and abnormal optical flow in a moving object / occlusion area. The threshold judgment condition corresponding to the abnormal optical flow in the flat area is the first condition, and the threshold judgment condition corresponding to the abnormal optical flow in the moving object / occlusion area is the second condition. Assuming that the image gradient information and the optical flow gradient information of the superpixel edge satisfy the first condition, it can be determined that the optical flow covered by the superpixel edge belongs to the abnormal optical flow in the flat area; assuming that the image gradient information and the optical flow gradient information of the superpixel edge satisfy the second condition, it can be determined that the optical flow covered by the superpixel edge belongs to the abnormal optical flow in the moving object / occlusion area. If the image gradient information and the optical flow gradient information of the superpixel edge neither satisfy the first condition nor the second condition, it can be determined that the optical flow covered by the superpixel edge is normal optical flow and not abnormal optical flow.

[0107] In some embodiments, it can be determined that there is abnormal optical flow in the optical flow information based on the image gradient information and the optical flow gradient information of the superpixel edge; the type of the abnormal optical flow can be determined based on the image gradient information and the optical flow gradient information of the superpixel edge. That is to say, when it is determined that there is abnormal optical flow in the optical flow information based on the image gradient information and the optical flow gradient information of the superpixel edge, the type of the abnormal optical flow can be determined based on the image gradient information and the optical flow gradient information of the superpixel edge.

[0108] It should be noted that through the mutual verification of the image information and the optical flow information, the present disclosure screens the positions where incorrect optical flow is likely to occur, so that it can be applied to the detection of abnormal optical flow in various difficult scenarios such as weak texture, repetitive texture, and target occlusion, and improves the detection effect of abnormal optical flow. In some embodiments, the optional implementation manners of determining that there is abnormal optical flow in the optical flow information based on the image gradient information and the optical flow gradient information of the superpixel edge include at least one of the following steps 205a to 205c.

[0109] Step 205a, when the image gradient information of the superpixel edge is less than or equal to the first image gradient threshold and the optical flow gradient information of the superpixel edge is greater than or equal to the first optical flow gradient threshold, determine that the optical flow associated with the superpixel edge is abnormal optical flow.

[0110] Exemplarily, for weak texture or flat area texture, the image gradient information of the corresponding superpixel edge is usually less than or equal to the first image gradient threshold. For example, the image gradient information of the superpixel edge in the weak texture or flat area texture is usually close to 0, while the optical flow gradient information value at the corresponding position is very high, then it can be determined that the optical flow associated with the position is an abnormal optical flow.

[0111] Step 205b, when the image gradient information of the superpixel edge is greater than or equal to the first image gradient threshold, and / or the optical flow gradient information of the superpixel edge is less than or equal to the first optical flow gradient threshold, determine the image gradient information of the superpixel edge at the boundary of the target area, the image gradient information of the superpixel edge inside the target area, the optical flow gradient information of the superpixel edge at the boundary of the target area, and the optical flow gradient information of the superpixel edge inside the target area.

[0112] The target area may be a target area identified from a reference image using a target detection algorithm.

[0113] Step 205c, when the image gradient information of the superpixel edge at the boundary of the target area is greater than or equal to the second image gradient threshold, and the image gradient information of the superpixel edge inside the target area is less than or equal to the first image gradient threshold, and the optical flow gradient information of the superpixel edge at the boundary of the target area and the optical flow gradient information of the superpixel edge inside the target area are both greater than or equal to the first optical flow gradient threshold, it is determined that the optical flow associated with the target area is an abnormal optical flow; wherein the second image gradient threshold is greater than the first image gradient threshold.

[0114] For example, for occluded or moving areas, the image gradient information value of the superpixel edge at the boundary of the moving object is larger, while the image gradient information value of the superpixel edge inside the moving object is smaller; corresponding to the optical flow gradient image, the optical flow in the occluded area of ​​the moving object changes dramatically. If the optical flow gradient information value at the corresponding position is larger, the optical flow covered by the superpixel in this area is abnormal.

[0115] In order to facilitate those skilled in the art to more clearly understand the solutions of the present disclosure, examples are described below.

[0116] like Figure 3 As shown in the figure, the process of abnormal optical flow detection is described by taking image 1 and image 2 as examples. Among them, image 1 and image 2 are two images used to calculate the optical flow, and image 1 is a reference image (also called a reference frame). The optical flow algorithm can be used to calculate the optical flow information between image 1 and image 2. The superpixel segmentation algorithm is used to perform superpixel segmentation on image 1 to obtain superpixel information. After obtaining the superpixel information and optical flow information, the gradient of the image superpixel edge and the optical flow superpixel edge are calculated respectively to obtain the gradient information of the two, and the abnormal optical flow is screened according to the gradient information, thereby completing the abnormal optical flow detection.

[0117] Taking the Figure 4a shown image as an example, it is the superpixel segmentation result of a grayscale image, and the white edges are the edges between different superpixels. For the original grayscale image, according to the Figure 4a shown superpixel edge positions, calculate the pixel change gradient of the grayscale image at these positions. Assuming the position coordinates are (x, y) and the pixel value is g(x, y), then sum the gradients in the horizontal, vertical, and diagonal directions to obtain the image gradient information grad_g(x, y) at this position. For non-superpixel edge regions, set their gradient values to 0. As Figure 4b shown, it is an example diagram of the visualization of the superpixel edge gradient after calculating the gradient information for the Figure 4a shown image. Among them, Figure 4b the white edges in it can be considered to have large differences in texture, color, brightness, etc. from the surrounding pixel points, so as to more prominently show the semantic information characteristics of the superpixels.

[0118] For the optical flow information (also called the optical flow map), calculate the optical flow change gradient at the superpixel edge positions. Assuming the position coordinates are (x, y) and the optical flow value is mv(x, y), then sum the gradients in the horizontal, vertical, and diagonal directions to obtain the optical flow gradient information grad_mv(x, y) at this position. For non-superpixel edge regions, set their optical flow gradient values to 0. As Figure 5a shown, it is the optical flow map between Image 1 and Image 2. After calculating the gradient information for the Figure 5a shown optical flow information, the visualization result of the superpixel edge gradient of this optical flow map can be as Figure 5b shown. Among them, Figure 5b the white edges in it can be considered to have large changes in the superpixel edge gradient.

[0119] After obtaining the image gradient information and optical flow gradient information of the superpixel edges, the optical flow quality can be evaluated at the superpixel scale according to the image gradient information and optical flow gradient information of the superpixel edges. For weak texture or flat area textures, for example, as Figure 6a the area A in it is the superpixel segmentation result of the original image, Figure 6b the area B in it is the optical flow result at the corresponding position (for the original image, area A and area B are the same area, such as containing the same pixel points). The superpixel edge gradient corresponding to area A is as Figure 4b shown, the superpixel edge gradient corresponding to area B is as Figure 5b shown. Area A is a flat area, and the image gradient information grad_g in this flat area is close to 0, while the optical flow gradient value in area B is very high. Then the optical flow covered by this superpixel is abnormal optical flow, that is, Figure 6b the optical flow in area B in it is abnormal optical flow.

[0120] For occluded or moving areas, such as Figure 7a - Figure 7d As shown in the figure, the image gradient information grad_g at the boundary of the moving object C has a larger value, while the image gradient information (such as grad_g”) at the superpixel boundary inside the moving object C has a smaller value; corresponding to the optical flow gradient image, the optical flow of the moving object occlusion area D changes dramatically, and the grad_mv value is large, then the optical flow covered by the superpixel in this area is abnormal. For example, Figure 8 As shown, the process of screening abnormal optical flow based on the image gradient information and optical flow gradient information of the superpixel edge can be as follows: when the image gradient information grad_g of the superpixel edge is less than thres_g1 (such as the first image gradient threshold value mentioned above) and the optical flow gradient information grad_mv of the superpixel edge is greater than thres_mv (such as the first optical flow gradient threshold value mentioned above), determine that the optical flow covered by the superpixel edge is an abnormal optical flow in the flat area. When the image gradient information grad_g of the superpixel edge is not less than thres_g1 (such as the first image gradient threshold value mentioned above), and / or the optical flow gradient information grad_mv of the superpixel edge is not greater than thres_mv (such as the first optical flow gradient threshold value mentioned above), determine the image gradient information of the superpixel edge at the boundary of the target area, the image gradient information of the superpixel edge inside the target area, the optical flow gradient information of the superpixel edge at the boundary of the target area, and the optical flow gradient information of the superpixel edge inside the target area. When the image gradient information grad_g of the superpixel edge at the boundary of the target area is greater than thres_g2 (such as the second image gradient threshold value mentioned above), and the image gradient information grad_g of the superpixel edge inside the target area is greater than thres_g3 (such as the second image gradient threshold value mentioned above), and the image gradient information grad_g of the superpixel edge inside the target area is greater than thres_g4 (such as the second image gradient threshold value mentioned above), and the image gradient information grad_g of the superpixel edge inside the target area is greater than thres_g5 (such as the second image gradient threshold value mentioned above). ′′ If it is less than thres_g1, and the optical flow gradient information of the superpixel edge at the boundary of the target area and the optical flow gradient information of the superpixel edge inside the target area are both greater than thres_mv, it can be determined that the optical flow associated with the target area is an abnormal optical flow.

[0121] In summary, the present invention uses the image information and optical flow information of the superpixel edge to screen the abnormal optical flow. Different from the abnormal optical flow detection method in the related art, there is no need to calculate the optical flow twice, nor to warp the image. Instead, the abnormal optical flow detection is completed by extracting the information of the superpixel edge, which greatly reduces the computational cost. At the same time, the present invention mainly focuses on optical flow detection in difficult scenes such as large-scale motion, weak texture, repeated texture, target occlusion, etc. The screening of abnormal optical flow is more direct, ensuring the detection effect of erroneous optical flow.

[0122] Figure 9 FIG. 1 is a block diagram of an abnormal optical flow detection device according to an exemplary embodiment. Figure 9, the device includes a first determination module 910, a second determination module 920, a third determination module 930, and a detection module 940.

[0123] Among them, the first determination module 910 is used to determine the optical flow information between the first image and the second image.

[0124] The second determination module 920 is used to determine the superpixel information of the reference image, and the superpixel information is the superpixel segmentation result obtained by performing superpixel segmentation on the reference image; the reference image is the first image or the second image.

[0125] The third determination module 930 is used to determine the image gradient information and the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information. In some embodiments, the third determination module 930 includes a first determination unit and a second determination unit. Among them, the first determination unit is used to determine the image gradient information of the superpixel edge according to the superpixel information and the pixel information of the reference image; the second determination unit is used to determine the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information.

[0126] In some embodiments, the first determination unit is specifically used for: according to the superpixel information, determining the pixel values of the pixels around the superpixel edge in the reference image; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; according to the pixel values of the pixels around the superpixel edge, statistically calculating the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions to obtain the image gradient information of the superpixel edge.

[0127] In some embodiments, the second determination unit is specifically used for: according to the superpixel information, determining the optical flow values of the pixels around the superpixel edge in the optical flow information; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; according to the optical flow values of the pixels around the superpixel edge, statistically calculating the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions to obtain the optical flow gradient information of the superpixel edge.

[0128] The detection module 940 is used to detect whether there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge.

[0129] In some embodiments, the detection module is used to perform at least one of the following: determining that there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge; determining the type of abnormal optical flow according to the image gradient information and the optical flow gradient information of the superpixel edge.

[0130] In some embodiments in combination with some embodiments of the second aspect, the detection module is configured to perform at least one of the following: determining that the optical flow associated with the superpixel edge is abnormal optical flow when the image gradient information of the superpixel edge is less than or equal to the first image gradient threshold and the optical flow gradient information of the superpixel edge is greater than or equal to the first optical flow gradient threshold; determining the image gradient information of the superpixel edge at the boundary of the target region, the image gradient information of the superpixel edge inside the target region, the optical flow gradient information of the superpixel edge at the boundary of the target region, and the optical flow gradient information of the superpixel edge inside the target region when the image gradient information of the superpixel edge is greater than or equal to the first image gradient threshold, and / or the optical flow gradient information of the superpixel edge is less than or equal to the first optical flow gradient threshold; the target region is a target region identified from the reference image by using a target detection algorithm; determining that the optical flow associated with the target region is abnormal optical flow when the image gradient information of the superpixel edge at the boundary of the target region is greater than or equal to the second image gradient threshold, the image gradient information of the superpixel edge inside the target region is less than or equal to the first image gradient threshold, and the optical flow gradient information of the superpixel edge at the boundary of the target region and the optical flow gradient information of the superpixel edge inside the target region are both greater than or equal to the first optical flow gradient threshold; wherein the second image gradient threshold is greater than the first image gradient threshold.

[0131] In some embodiments, the apparatus further includes: a fourth determination module configured to: determine the image gradient value of the non-superpixel edge region in the reference image as the first value; and / or determine the optical flow gradient value of the non-superpixel edge region among the optical flow information as the first value.

[0132] Regarding the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0133] Figure 10 It is a schematic structural diagram of a communication device 1000 proposed in an embodiment of the present disclosure. The communication device 1000 may be a network device (such as an access network device, a core network device, etc.), or a terminal (such as a user equipment, etc.), or a chip, a chip system, or a processor, etc. that supports the network device to implement any of the above methods, or a chip, a chip system, or a processor, etc. that supports the terminal to implement any of the above methods. The communication device 1000 can be used to implement the method described in the above method embodiments, and specific reference can be made to the description in the above method embodiments.

[0134] As Figure 10As shown, the communication device 1000 includes one or more processors 1001. The processor 1001 can be a general-purpose processor or a dedicated processor, etc. For example, it can be a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, and the central processing unit can be used to control a communication device (such as a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute programs, and process program data. Optionally, the communication device 1000 is used to execute any of the above methods. Optionally, one or more processors 1001 are used to call instructions to cause the communication device 1000 to execute any of the above methods.

[0135] In some embodiments, the communication device 1000 further includes one or more transceivers 1002. When the communication device 1000 includes one or more transceivers 1002, the transceiver 1002 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processor 1001 performs at least one of the other steps (such as steps 101 to 104, steps 201 to 205, but not limited thereto). In an alternative embodiment, the transceiver may include a receiver and / or a transmitter, and the receiver and the transmitter may be separate or integrated together. Optionally, terms such as transceiver, transceiver unit, transceiver machine, transceiver circuit, interface circuit, interface, etc. can be replaced with each other, terms such as transmitter, transmitter unit, transmitter machine, transmitter circuit, etc. can be replaced with each other, and terms such as receiver, receiver unit, receiver machine, receiver circuit, etc. can be replaced with each other.

[0136] In some embodiments, the communication device 1000 further includes one or more memories 1003 for storing data. Optionally, all or part of the memories 1003 may also be outside the communication device 1000. In an alternative embodiment, the communication device 1000 may include one or more interface circuits 1004. Optionally, the interface circuit 1004 is connected to the memory 1002, and the interface circuit 1004 can be used to receive data from the memory 1002 or other devices, and can be used to send data to the memory 1002 or other devices. For example, the interface circuit 1004 can read the data stored in the memory 1002 and send the data to the processor 1001.

[0137] The communication device 1000 described in the above embodiments can be a network device or a terminal, but the scope of the communication device 1000 described in this disclosure is not limited thereto, and the structure of the communication device 1000 can be unrestricted Figure 10Limitations. The communication device can be an independent device or can be part of a larger device. For example, the communication device can be: 1) an independent integrated circuit (IC), or chip, or system-on-chip or subsystem; (2) a set of one or more ICs, optionally, the above IC set can also include storage components for storing data and programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, a smart terminal device, a cellular phone, a wireless device, a handset, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) others, etc.

[0138] Figure 11 is a schematic structural diagram of the chip 1100 proposed in the embodiments of the present disclosure. For the case where the communication device 1000 can be a chip or a system-on-chip, reference can be made to Figure 11 the schematic structural diagram of the chip 1100 shown, but not limited thereto.

[0139] The chip 1100 includes one or more processors 1101. The chip 1100 is used to execute any of the above methods.

[0140] In some embodiments, the chip 1100 further includes one or more interface circuits 1102. Optionally, terms such as interface circuit, interface, and transceiver pin can be replaced with each other. In some embodiments, the chip 1100 further includes one or more memories 1103 for storing data. Optionally, all or part of the memories 1103 can be outside the chip 1100. Optionally, the interface circuit 1102 is connected to the memory 1103. The interface circuit 1102 can be used to receive data from the memory 1103 or other devices, and the interface circuit 1102 can be used to send data to the memory 1103 or other devices. For example, the interface circuit 1102 can read the data stored in the memory 1103 and send the data to the processor 1101.

[0141] In some embodiments, the interface circuit 1102 executes at least one of the communication steps such as sending and / or receiving in the above method. The interface circuit 1102 executing the communication steps such as sending and / or receiving in the above method means, for example, that the interface circuit 1102 executes data interaction between the processor 1101, the chip 1100, the memory 1103, or the transceiver device. In some embodiments, the processor 1101 executes at least one of the other steps (such as steps 101 to 104, steps 201 to 205, but not limited thereto).

[0142] In various embodiments such as virtual devices, physical devices, and chips, the various modules and / or components described can be combined or separated arbitrarily according to circumstances. Optionally, some or all of the steps can also be executed collaboratively by multiple modules and / or components, and no limitation is imposed here.

[0143] The present disclosure also provides a storage medium. Instructions are stored on the above storage medium. When the above instructions run on the communication device 1000, the communication device 1000 is caused to execute any of the above methods. Optionally, the above storage medium is an electronic storage medium. Optionally, the above storage medium is a computer-readable storage medium, but not limited thereto, and it can also be other device-readable storage mediums. Optionally, the above storage medium can be a non-transitory storage medium, but not limited thereto, and it can also be a transitory storage medium.

[0144] The present disclosure also provides a program product. When the above program product is executed by the communication device 1000, the communication device 1000 is caused to execute any of the above methods. Optionally, the above program product is a computer program product.

[0145] The present disclosure also provides a computer program. When it runs on a computer, the computer is caused to execute any of the above methods.

[0146] Those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present invention are pointed out by the following claims.

[0147] It should be understood that the present invention is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An abnormal optical flow detection method, characterized in that, it includes: Determine the optical flow information between the first image and the second image; Determine the superpixel information of the reference image, where the superpixel information is the superpixel segmentation result obtained by performing superpixel segmentation on the reference image; The reference image is the first image or the second image; According to the superpixel information and the optical flow information, determine the image gradient information and the optical flow gradient information of the superpixel edge; According to the image gradient information and the optical flow gradient information of the superpixel edge, detect whether there is abnormal optical flow in the optical flow information.

2. The method according to claim 1, characterized in that, The step of determining the image gradient information and the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information includes: Determine the image gradient information of the superpixel edge according to the superpixel information and the pixel information of the reference image; Determine the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information.

3. The method according to claim 2, characterized in that, The step of determining the image gradient information of the superpixel edge according to the superpixel information and the pixel information of the reference image includes: According to the superpixel information, determine the pixel values of the pixels around the superpixel edge in the reference image; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; According to the pixel values of the pixels around the superpixel edge, calculate the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions to obtain the image gradient information of the superpixel edge.

4. The method according to claim 2, characterized in that, The step of determining the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information includes: According to the superpixel information, determine the optical flow values of the pixels around the superpixel edge in the optical flow information; the pixels around the superpixel edge include the pixels adjacent to the superpixel edge in the horizontal, vertical, and diagonal directions; According to the optical flow values of the pixels around the superpixel edge, calculate the sum of the gradients of the superpixel edge in the horizontal, vertical, and diagonal directions to obtain the optical flow gradient information of the superpixel edge.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Determine the image gradient value of the non-superpixel edge region in the reference image as the first value; and / or, Determine the optical flow gradient value of the non-superpixel edge region in the optical flow information as the first value.

6. The method according to claim 1, characterized in that, The step of detecting whether there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge includes at least one of the following: Determine that there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge; Determine the type of the abnormal optical flow according to the image gradient information and the optical flow gradient information of the superpixel edge.

7. The method according to claim 6, characterized in that, Determining that there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge includes at least one of the following: When the image gradient information of the superpixel edge is less than or equal to a first image gradient threshold and the optical flow gradient information of the superpixel edge is greater than or equal to a first optical flow gradient threshold, determining that the optical flow associated with the superpixel edge is abnormal optical flow; When the image gradient information of the superpixel edge is greater than or equal to the first image gradient threshold, and / or, the optical flow gradient information of the superpixel edge is less than or equal to the first optical flow gradient threshold, determining the image gradient information of the superpixel edge at the boundary of the target region, the image gradient information of the superpixel edge inside the target region, the optical flow gradient information of the superpixel edge at the boundary of the target region, and the optical flow gradient information of the superpixel edge inside the target region; The target region is a target region identified from the reference image by using a target detection algorithm; When the image gradient information of the superpixel edge at the boundary of the target region is greater than or equal to a second image gradient threshold, the image gradient information of the superpixel edge inside the target region is less than or equal to the first image gradient threshold, and the optical flow gradient information of the superpixel edge at the boundary of the target region and the optical flow gradient information of the superpixel edge inside the target region are both greater than or equal to the first optical flow gradient threshold, determining that the optical flow associated with the target region is abnormal optical flow; wherein, the second image gradient threshold is greater than the first image gradient threshold.

8. An abnormal optical flow detection device Characterized in that It includes: A first determination module, configured to determine the optical flow information between a first image and a second image; A second determination module, configured to determine the superpixel information of a reference image, where the superpixel information is a superpixel segmentation result obtained by performing superpixel segmentation on the reference image; The reference image is the first image or the second image; A third determination module, configured to determine the image gradient information and the optical flow gradient information of the superpixel edge according to the superpixel information and the optical flow information; A detection module, configured to detect whether there is abnormal optical flow in the optical flow information according to the image gradient information and the optical flow gradient information of the superpixel edge.

9. A communication device Characterized in that It includes: One or more processors; Wherein, the processor is configured to call instructions to cause the communication device to execute the abnormal optical flow detection method according to any one of claims 1-7.

10. A storage medium, the storage medium stores instructions Characterized in that When the instructions run on the communication device, the communication device is caused to execute the abnormal optical flow detection method according to any one of claims 1-7.

Images