Video stream filtering method, apparatus, device, and storage medium

Abstract

The application discloses a video stream filtering method and device, equipment and a storage medium, and the method comprises the steps that three-dimensional space information of a two-dimensional feature point is acquired; the two-dimensional feature point is determined from a frame image of a video stream; a normal projection value of the two-dimensional feature point is determined based on the three-dimensional space information; the normal projection value is input to a three-dimensional filtering core to obtain a filtering result; and the three-dimensional filtering core is used for filtering the video stream based on the normal projection value. The application realizes filtering of the video stream through three-dimensional information, so that the boundary information determined thereby is three-dimensional boundary information, and the problem of missing three-dimensional boundary information does not exist, thereby improving the accuracy of the boundary information obtained by filtering the video stream.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video stream processing, in particular to a video stream filtering method and device, equipment and storage medium. BACKGROUND

[0002] AR(Augmented Reality, augmented reality) needs to use three-dimensional modeling when using. Three-dimensional modeling is mainly realized based on boundary detection and processing technology.

[0003] Video stream filtering is a boundary detection and processing technology. The current video stream filtering considers the boundary information of the intra-neighbor pixels of a single frame image and the inter-frame information of adjacent frames. The information considered is two-dimensional boundary information in the image (including light mutation, color mutation, etc.). Therefore, three-dimensional modeling based on two-dimensional boundary information will cause the loss of three-dimensional boundary information, so that the three-dimensional object obtained by three-dimensional modeling based on two-dimensional boundary information is not accurate enough. That is, since the boundary information is two-dimensional, the boundary information obtained by video stream filtering is not accurate enough for three-dimensional modeling. SUMMARY

[0004] Therefore, the embodiments of the present application provide a video stream filtering method, device, equipment and storage medium, aiming to improve the accuracy of the boundary information obtained by video stream filtering.

[0005] To achieve the above object, the present application provides a video stream filtering method, which comprises:

[0006] obtaining three-dimensional space information of a two-dimensional feature point; the two-dimensional feature point is determined from a frame image of a video stream;

[0007] determining a normal projection value of the two-dimensional feature point based on the three-dimensional space information;

[0008] inputting the normal projection value to a three-dimensional filtering kernel to obtain a filtering result; the three-dimensional filtering kernel is used to filter the video stream based on the normal projection value.

[0009] Illustratively, the inputting the normal projection value to the three-dimensional filtering kernel to obtain the filtering result comprises:

[0010] obtaining a weight of a neighbor pixel of the two-dimensional feature point;

[0011] inputting the weight and the normal projection value to the three-dimensional filtering kernel to obtain the filtering result; the three-dimensional filtering kernel at least includes a two-dimensional filtering kernel, and the two-dimensional filtering kernel is a pixel value of the neighbor pixel.

[0012] Illustratively, the obtaining the three-dimensional space information of the two-dimensional feature point comprises:

[0013] reconstructing the two-dimensional feature point to obtain a three-dimensional coordinate corresponding to the two-dimensional feature point; the three-dimensional coordinate is three-dimensional space information of the two-dimensional feature point.

[0014] For example, the two-dimensional feature point is reconstructed to obtain a three-dimensional coordinate corresponding to the two-dimensional feature point, which includes:

[0015] Obtain the first two-dimensional coordinate of the two-dimensional feature point on the image;

[0016] Obtain two adjacent images which are adjacent frames to the image;

[0017] Obtain two second two-dimensional coordinates of two adjacent two-dimensional feature points on two adjacent images; the two adjacent two-dimensional feature points are obtained by imaging the same three-dimensional feature point as the two-dimensional feature point;

[0018] Based on the first two-dimensional coordinate and the two second two-dimensional coordinates, the three-dimensional coordinate is calculated.

[0019] For example, the three-dimensional coordinate is calculated based on the first two-dimensional coordinate and the at least two second two-dimensional coordinates, which includes:

[0020] Input the first two-dimensional coordinate and the at least two second two-dimensional coordinates into a coordinate conversion model to obtain a three-dimensional coordinate; the coordinate conversion model is constructed based on the imaging principle of the camera.

[0021] For example, the normal projection value of the two-dimensional feature point is determined based on the three-dimensional space information, which includes:

[0022] Project the three-dimensional feature point to the image to obtain the normal projection value of the two-dimensional feature point.

[0023] For example, the three-dimensional feature point is projected to the image to obtain the normal projection value of the two-dimensional feature point, which includes:

[0024] Obtain the normal of the face sheet corresponding to the three-dimensional feature point, and normalize the normal;

[0025] After projecting the three-dimensional feature point to the image, the normal projection value of the two-dimensional feature point is determined based on the angle between the normalized normal and the plane on which the image is located.

[0026] In addition, in order to achieve the above purpose, the application also provides a video stream filtering device, the device comprises:

[0027] The acquisition module is used for acquiring three-dimensional space information of a two-dimensional feature point; the two-dimensional feature point is determined from one frame of image of a video stream;

[0028] determining a normal projection value of the two-dimensional feature point based on the three-dimensional spatial information;

[0029] filtering the video stream based on the normal projection value.

[0030] In an example, the filtering module is specifically configured to:

[0031] obtain a weight of a neighborhood pixel of the two-dimensional feature point;

[0032] input the weight and the normal projection value to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel at least includes a two-dimensional filter kernel, and the two-dimensional filter kernel is a pixel value of the neighborhood pixel.

[0033] In an example, the obtaining module is specifically configured to:

[0034] perform visual reconstruction on the two-dimensional feature point to obtain a three-dimensional coordinate corresponding to the two-dimensional feature point; the three-dimensional coordinate is three-dimensional spatial information of the two-dimensional feature point.

[0035] In an example, the obtaining module is further configured to:

[0036] obtain a first two-dimensional coordinate of the two-dimensional feature point on the image;

[0037] obtain two adjacent images which are adjacent frames to the image;

[0038] obtain two second two-dimensional coordinates of two adjacent two-dimensional feature points on the two adjacent images; the two adjacent two-dimensional feature points and the two-dimensional feature point are obtained by imaging of a same three-dimensional feature point;

[0039] calculate the three-dimensional coordinate based on the first two-dimensional coordinate and the two second two-dimensional coordinates.

[0040] In an example, the obtaining module is further configured to:

[0041] input the first two-dimensional coordinate and the at least two second two-dimensional coordinates to a coordinate conversion model to obtain a three-dimensional coordinate; the coordinate conversion model is constructed based on an imaging principle of a camera.

[0042] In an example, the determining module is specifically configured to:

[0043] project the three-dimensional feature point to the image to obtain a normal projection value of the two-dimensional feature point.

[0044] In an example, the determining module is specifically configured to:

[0045] obtaining a normal of a face sheet corresponding to the three-dimensional feature point, and performing normalization processing on the normal;

[0046] After projecting the three-dimensional feature point to the image, determining a normal projection value of the two-dimensional feature point based on an included angle between the normalized normal and a plane on which the image is located.

[0047] In addition, to achieve the above object, the present application also provides a video stream filtering device, which comprises a memory, a processor and a video stream filtering program stored in the memory and executable on the processor, and the video stream filtering program implements the steps of the video stream filtering method when executed by the processor.

[0048] In addition, to achieve the above object, the present application also provides a computer readable storage medium, which stores a video stream filtering program, and the video stream filtering program implements the steps of the video stream filtering method when executed by a processor.

[0049] Compared with the prior art in which the boundary information is two-dimensional, for three-dimensional modeling, due to the lack of three-dimensional boundary information, the boundary information obtained by video stream filtering is not accurate enough. In the present application, three-dimensional space information of a two-dimensional feature point is obtained; the two-dimensional feature point is determined from a frame of image of a video stream; based on the three-dimensional space information, a normal projection value of the two-dimensional feature point is determined; the normal projection value is input to a three-dimensional filtering kernel to obtain a filtering result; the three-dimensional filtering kernel is used to filter the video stream based on the normal projection value. In the present application, three-dimensional information, i.e. the normal projection value, is considered when filtering the video stream, so that the video stream is filtered by three-dimensional information, specifically by a three-dimensional filtering kernel, so that the boundary information determined thereby is boundary information containing three dimensions, and the problem of lack of three-dimensional boundary information does not exist, thereby improving the accuracy of the boundary information obtained by video stream filtering. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a flowchart of a first embodiment of the video stream filtering method of the present application;

[0051] Figure 2 a is an imaging principle schematic diagram of a camera involved in an embodiment of the video stream filtering method of the present application, Figure 2 b is a schematic diagram of the correspondence between an image and a real environment involved in an embodiment of the video stream filtering method of the present application;

[0052] Figure 3 is an imaging schematic diagram of a camera in two postures in the first embodiment of the video stream filtering method of the present application;

[0053] Figure 4 is a neighborhood diagram of the diamond neighborhood weighting method in the first embodiment of the video stream filtering method of the present application;

[0054] Figure 5 is a structural diagram of a hardware running environment involved in the embodiment of the present application.

[0055] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0057] The present application provides a video stream filtering method, referring to Figure 1 , Figure 1 is a flow diagram of the first embodiment of the video stream filtering method of the present application.

[0058] The embodiments of the video stream filtering method are provided in the present application, and it should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be performed in an order different from that shown herein. The video stream filtering method can be applied in a server or a terminal.

[0059] For the convenience of description, the following omits the description of the execution subject of each step of the video stream filtering method. The video stream filtering method comprises:

[0060] Step S10, obtaining three-dimensional space information of a two-dimensional feature point; the two-dimensional feature point is determined from a frame of image of a video stream.

[0061] For example, the process of determining the two-dimensional feature point from the frame of image of the video stream can be realized by an image feature extraction technology with scale invariance and rotation invariance, including shift, orb, etc. It can be understood that the two-dimensional feature point is a key point in the image, for example, a point representing the outline of the mouth, nose, etc. in a face image.

[0062] It should be noted that the two-dimensional feature point is a pixel on the boundary of the object in the image, i.e. the boundary information.

[0063] For example, the obtaining of the three-dimensional space information of the two-dimensional feature point comprises:

[0064] Step a, performing visual reconstruction on the two-dimensional feature point to obtain a three-dimensional coordinate corresponding to the two-dimensional feature point; the three-dimensional coordinate is the three-dimensional space information of the two-dimensional feature point.

[0065] An image (two-dimensional) is obtained by imaging a real environment (three-dimensional) in a camera, referring to Figure 2 a and Figure 2 b, Figure 2 a is the imaging principle of the camera, Figure 2 b is the proportional relationship between the image and the real environment. Wherein, the spatial coordinate origin C is the center of the camera (monocular camera), P is the pixel origin of the image, X is a three-dimensional feature point, and x is the image formed by X in the image (i.e. a two-dimensional feature point). It should be noted that the process of visual reconstruction is the process of reconstructing the real environment based on the image, that is, the inverse imaging process. The three-dimensional feature point is restored by the visual reconstruction of the two-dimensional feature point, so that the three-dimensional coordinates obtained are more accurate.

[0066] For example, the two-dimensional feature points are visually reconstructed to obtain the three-dimensional coordinates corresponding to the two-dimensional feature points, comprising:

[0067] Step a1, obtaining the first two-dimensional coordinates of the two-dimensional feature points on the image.

[0068] The coordinate origin p(p x ,p x ) of the image is generally the center of the image, that is, the first two-dimensional coordinates (u1, v1) are the coordinates of the two-dimensional feature points in the two-dimensional coordinate system with p(p x ,p y ) as the coordinate origin. The coordinate origin p(p x ,p y ) of the image can also be other positions of the image, such as the four corners of the image.

[0069] Step a2, obtaining two adjacent images which are adjacent to each other with the image.

[0070] Adjacent frames are the previous frame and the next frame of the image in the video stream.

[0071] Step a3, obtaining two second two-dimensional coordinates of two adjacent two-dimensional feature points on two adjacent images; the two adjacent two-dimensional feature points are obtained by imaging the same three-dimensional feature point as the two-dimensional feature point.

[0072] The specific implementation of the two second two-dimensional coordinates (u2, v2) and (u3, v3) is basically the same as that of the above-mentioned first two-dimensional coordinates, which will not be repeated here.

[0073] Referring to Figure 3 , Figure 3Fig. 1 is a schematic diagram of imaging of a three-dimensional feature point P by a camera in two poses. In the diagram, the two images of the three-dimensional feature point P are p and p', and the camera center O2 is obtained by rotating and translating the camera center O1 by a rotation matrix R and a translation matrix T. It can be understood that the camera pose can be changed by rotating and translating the camera pose by the rotation matrix R and the translation matrix T.

[0074] Step a4, calculating the three-dimensional coordinates based on the first two-dimensional coordinates and the two second two-dimensional coordinates.

[0075] The camera imaging principle is x = KX, where K is a camera parameter, and the camera parameter includes camera intrinsic parameters and camera extrinsic parameters, where,

[0076]

[0077] where s is a scale factor (used to adjust the image scaling ratio of the camera), f x , f y , p x , p y , and γ are camera intrinsic parameters, f x and f y represent focal lengths, p x and p y are parameters of the coordinate origin p (p x , p x ), and γ represents a coordinate axis tilt parameter; α1, α2, and t are camera extrinsic parameters, α1 and α2 represent a rotation vector R, and t represents a translation vector T.

[0078] In the imaging process, X (x, y, z) is converted into x (u i , v i ) by K, and the conversion process is realized by the following formula:

[0079]

[0080] Thus, three equations are obtained, that is,

[0081] and Solving the three equations together, the three-dimensional coordinates X (x, y, z) can be obtained.

[0082] Illustratively, the calculating the three-dimensional coordinates based on the first two-dimensional coordinates and the at least two second two-dimensional coordinates includes:

[0083] Step a41, inputting the first two-dimensional coordinates and the at least two second two-dimensional coordinates into a coordinate conversion model to obtain three-dimensional coordinates; the coordinate conversion model is constructed based on the imaging principle of the camera.

[0084] The coordinate conversion model is constructed by the formula x=KX, that is, after inputting the first two-dimensional coordinate and at least two second two-dimensional coordinates into the coordinate conversion model, the coordinate conversion model can automatically calculate the three-dimensional coordinate. It can be understood that the calculation efficiency can be improved by using a fixed calculation model.

[0085] In step S20, a normal projection value of the two-dimensional feature point is determined based on the three-dimensional space information.

[0086] For example, the determination of the normal projection value of the two-dimensional feature point based on the three-dimensional space information includes:

[0087] In step b, the three-dimensional feature point is projected to the image to obtain the normal projection value of the two-dimensional feature point.

[0088] It should be noted that there are multiple two-dimensional feature points in the image, and the determination process of each two-dimensional feature point and its corresponding three-dimensional coordinate is basically the same, which will not be repeated here.

[0089] The process of projecting the three-dimensional feature point to the image is to project the face sheet corresponding to the three-dimensional feature point to the image. Therefore, the two-dimensional feature point is actually not one-to-one corresponding to the three-dimensional feature point, but one-to-one corresponding to the face sheet. The face sheet is composed of at least three three-dimensional feature points. The normal projection value is the size of the normal of the face sheet on the projection plane (i.e. the image).

[0090] For example, the projection of the three-dimensional feature point to the image to obtain the normal projection value of the two-dimensional feature point includes:

[0091] In step b1, the normal of the face sheet corresponding to the three-dimensional feature point is obtained, and the normal is normalized.

[0092] In step b2, after projecting the three-dimensional feature point to the image, the normal projection value of the two-dimensional feature point is determined based on the angle between the normalized normal and the plane on which the image is located.

[0093] Each face sheet has a normal perpendicular to the face sheet. The normal is normalized, and then the component of the normalized normal perpendicular to the image is determined according to the angle between the normalized normal corresponding to each face sheet and the normal and the plane on which the image is located, to obtain the normal projection value. It should be noted that the normal projection value of the face sheet parallel to the plane on which the image is located is the largest, and the normal projection value of the face sheet perpendicular to the plane on which the image is located is the smallest.

[0094] It should be noted that the normal projection value of each two-dimensional feature point in the image can determine whether the change of the normal is obvious (projection boundary effect, i.e. the light or color mutation is obvious on the boundary of the object, and whether the light or color mutation can be determined by whether the pixel value is mutated), for example, the normal projection value of a two-dimensional feature point is 0.8, the normal projection value of a two-dimensional feature point adjacent to it is 0.4, and the normal projection values of other two-dimensional feature points adjacent to it are similar to 0.8 (for example, 0.78, 0.82, etc.). Thus, it can be determined that the two-dimensional feature points with normal projection values of 0.4 and 0.8 are boundaries. Compared with only considering the two-dimensional boundary information (including light mutation, color mutation, etc.), if the light or color of the two-dimensional feature point does not change, it may be misjudged as not being a boundary.

[0095] In step S30, the normal projection value is input to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel is used to filter the video stream based on the normal projection value.

[0096] Since the normal projection value is determined by the three-dimensional feature point (three-dimensional boundary information), filtering the video stream by the normal projection value considers the three-dimensional boundary information.

[0097] For example, the input of the normal projection value to the three-dimensional filter kernel to obtain the filtering result includes:

[0098] In step c, the weight of the neighborhood pixel of the two-dimensional feature point is obtained.

[0099] In this embodiment, the diamond neighborhood weighting method is used to filter the video stream, instead of filtering all pixels in the image as neighborhood pixels, thereby improving the filtering speed. In addition, the square neighborhood weighting method or the like can also be used to filter the video stream.

[0100] The neighborhood pixel is a pixel adjacent to the two-dimensional feature point, and the adjacency is within a domain.

[0101] On the basis of considering the normal projection value, the contribution of the neighborhood pixel to the filtering process can also be considered. Referring to Figure 4 , Figure 4 The diamond neighborhood of the two-dimensional feature point is a domain adjacent to the two-dimensional feature point, which is a diamond. Each rectangle represents a pixel, and the number in the rectangle is the weight. It can be understood that the closer the neighborhood pixel is to the two-dimensional feature point, the greater the contribution and the greater the corresponding weight; the farther the neighborhood pixel is from the two-dimensional feature point, the smaller the contribution and the smaller the corresponding weight.

[0102] It should be noted that the range of the neighborhood is adjustable, for example Figure 4The diagonal line in the above formula is 5 pixels, and in addition, the diagonal line can also be 6 pixels, 7 pixels, etc.

[0103] Step d, inputting the weight and the normal projection value to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel at least includes a two-dimensional filter kernel, and the two-dimensional filter kernel is a pixel value of the neighborhood pixels.

[0104] To further improve the accuracy of filtering the video stream, the three-dimensional boundary information can be combined with the existing two-dimensional filter kernel (considering two-dimensional boundary information).

[0105] It should be noted that for image filtering (i.e. each frame in the video stream), the two-dimensional filter kernel considers the illumination information of the adjacent pixels in the image. From the illumination intensity according to the illumination information, the boundary and smoothing information of the image (non-smooth two-dimensional feature points are noise points) are determined. The boundary information of the illumination includes physical boundary (light information) and color boundary (color information). The two-dimensional filter kernel can filter out noise points and retain and enhance the contrast of the boundary according to the neighborhood statistical information of the pixels in the image (for example, the weight of the pixels in the above diamond neighborhood). Correspondingly, for video stream filtering (i.e. filtering each frame in the video stream), the effect of filtering out noise points and retaining and enhancing the contrast of the boundary can also be achieved. Therefore, the combination of the two-dimensional filter kernel can make up for the filtering ability of the filter kernel considering only three-dimensional information, thereby improving the accuracy of filtering the video stream.

[0106] Specifically, the mathematical expression of the three-dimensional filter kernel in the embodiment is:

[0107]

[0108] Wherein, ij is the subscript of the two-dimensional feature point; f(x mn is the pixel value of the above-mentioned neighborhood pixels; mn is the subscript of the neighborhood pixels, that is Figure 4 the label of the rectangle with numerical values in the above formula; K 3d is the normal projection value of the two-dimensional feature point; K 2d is the weight of the two-dimensional feature point and the neighborhood pixels; W p is a normalization coefficient, which satisfies the equation:

[0109]

[0110] It should be noted that filtering the video stream is to filter each frame of image in the video stream, so as to complete the filtering of the video stream. It can be understood that the process of filtering each frame of image is basically the same as the process of filtering a frame of image, which will not be repeated here.

[0111] It should be noted that the augmented reality is to implement simulation processing on the basis of computer and other science and technology, superimpose virtual information content in the real world to effectively apply, and in this process can be perceived by human senses, so as to realize the sensory experience beyond reality. The augmented reality technology can not only effectively reflect the content of the real world, but also can make the virtual information content display, and these delicate contents complement and superimpose each other. After the overlap between the real environment and the virtual object, they can exist in the same picture and space at the same time. The three-dimensional modeling involved in the augmented reality is realized based on edge detection and processing technology. The edge detection is to find the edge features from the above-mentioned two-dimensional feature points, and the edge processing is to remove the noise points of the non-edge features in the two-dimensional feature points. The processes of edge detection and processing can be realized through the above-mentioned filter kernel.

[0112] Compared with the prior art in which the boundary information is two-dimensional, for three-dimensional modeling, due to the lack of three-dimensional boundary information, the boundary information obtained by filtering the video stream is not accurate enough. The present application obtains three-dimensional space information of two-dimensional feature points; the two-dimensional feature points are determined from a frame of image of the video stream; based on the three-dimensional space information, a normal projection value of the two-dimensional feature points is determined; the normal projection value is input to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel is used to filter the video stream based on the normal projection value. When filtering the video stream, the present application considers the three-dimensional information, i.e. the normal projection value, so as to realize filtering the video stream through three-dimensional information, specifically filtering the video stream through the three-dimensional filter kernel, so that the boundary information determined thereby is three-dimensional boundary information, and the problem of lack of three-dimensional boundary information does not exist, thereby improving the accuracy of the boundary information obtained by filtering the video stream.

[0113] In addition, the present application also provides a video stream filtering device, the device comprises:

[0114] The acquisition module is configured to acquire three-dimensional space information of two-dimensional feature points; the two-dimensional feature points are determined from a frame of image of the video stream;

[0115] The determination module is configured to determine a normal projection value of the two-dimensional feature points based on the three-dimensional space information;

[0116] The filtering module is configured to input the normal projection value to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel is used to filter the video stream based on the normal projection value.

[0117] For example, the filtering module is specifically configured to:

[0118] Obtain the weight of the neighborhood pixels of the two-dimensional feature points;

[0119] inputting the weight and the normal projection value into a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel at least comprises a two-dimensional filter kernel, and the two-dimensional filter kernel is a pixel value of the neighborhood pixel.

[0120] Illustratively, the acquisition module is specifically configured to:

[0121] visually reconstructing the two-dimensional feature point to obtain a three-dimensional coordinate corresponding to the two-dimensional feature point; the three-dimensional coordinate is three-dimensional space information of the two-dimensional feature point.

[0122] Illustratively, the acquisition module is further configured to:

[0123] acquiring a first two-dimensional coordinate of the two-dimensional feature point on the image;

[0124] acquiring two adjacent images which are mutually adjacent frames of the image;

[0125] acquiring two second two-dimensional coordinates of two adjacent two-dimensional feature points on the two adjacent images; the two adjacent two-dimensional feature points are obtained by imaging the same three-dimensional feature point as the two-dimensional feature point;

[0126] calculating the three-dimensional coordinate based on the first two-dimensional coordinate and the two second two-dimensional coordinates.

[0127] Illustratively, the acquisition module is further configured to:

[0128] inputting the first two-dimensional coordinate and the at least two second two-dimensional coordinates into a coordinate conversion model to obtain a three-dimensional coordinate; the coordinate conversion model is constructed based on imaging principles of a camera.

[0129] Illustratively, the determination module is specifically configured to:

[0130] projecting the three-dimensional feature point to the image to obtain a normal projection value of the two-dimensional feature point.

[0131] Illustratively, the determination module is specifically configured to:

[0132] acquiring a normal of a patch corresponding to the three-dimensional feature point, and performing normalization processing on the normal;

[0133] determining the normal projection value of the two-dimensional feature point based on an included angle between the normalized normal and a plane on which the image is located after projecting the three-dimensional feature point to the image.

[0134] The video stream filtering device specific implementation of the present application is basically the same as the above-mentioned video stream filtering method, and will not be repeated here.

[0135] Further, the present application also provides a video stream filtering device. Figure 5 As shown in Figure 5 is a structural diagram of a hardware running environment involved in an embodiment of the present application.

[0136] It should be noted that Figure 5 that is, a structural diagram of a hardware running environment of the video stream filtering device.

[0137] As shown in Figure 5 The video stream filtering device can include a processor 5001, such as a CPU, a memory 5005, a user interface 5003, a network interface 5004, and a communication bus 5002. The communication bus 5002 is used to realize the connection and communication between these components. The user interface 5003 can include a display screen (Display), an input unit such as a keyboard (Keyboard), and can also include a standard wired interface, a wireless interface. The network interface 5004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 5005 can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 5005 can also be a storage device independent of the aforementioned processor 5001.

[0138] Optionally, the video stream filtering device can also include RF (Radio Frequency, radio frequency) circuit, sensor, audio circuit, WiFi module, etc.

[0139] Those skilled in the art can understand that Figure 5 The structure of the video stream filtering device shown in the above description does not constitute a limitation on the video stream filtering device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0140] As shown in Figure 5 The memory 5005, as a computer storage medium, can include an operating system, a network communication module, a user interface module, and a video stream filtering program. The operating system is a program that manages and controls the hardware and software resources of the video stream filtering device, and supports the running of the video stream filtering program and other software or programs.

[0141] In the video stream filtering device shown in Figure 5 The user interface 5003 is mainly used to connect the terminal and communicate data with the terminal, such as receiving user signaling data sent by the terminal; the network interface 5004 is mainly used for background server and communicates data with the background server; the processor 5001 can be used to call the video stream filtering program stored in the memory 5005, and execute the steps of the video stream filtering method as described above.

[0142] The video stream filtering device embodiment of the present application is basically the same as the above-mentioned video stream filtering method embodiments, and thus will not be described here again.

[0143] In addition, the present application also provides a computer readable storage medium, and the computer readable storage medium stores a video stream filtering program. When the video stream filtering program is executed by a processor, the steps of the video stream filtering method described above are implemented.

[0144] The computer readable storage medium embodiment of the present application is basically the same as the above-mentioned video stream filtering method embodiments, and thus will not be described here again.

[0145] In addition, in order to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program. When the computer program is executed by a processor, the steps of the video stream filtering method described above are implemented.

[0146] The computer program product embodiment of the present application is basically the same as the above-mentioned video stream filtering method embodiments, and thus will not be described here again.

[0147] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent in such processes, methods, articles, or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article, or device including the element.

[0148] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, a device, or a network device) to execute the methods described in the various embodiments of the present application.

[0150] The above merely preferred embodiments of the present application and are not intended to limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of video stream filtering, characterized by, The method comprises: acquiring three-dimensional space information of a two-dimensional feature point; the two-dimensional feature point is determined from a frame of image of a video stream; determining a normal projection value of the two-dimensional feature point based on the three-dimensional space information; inputting the normal projection value to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel is used for filtering the video stream based on the normal projection value.

2. The method of claim 1, wherein, The inputting the normal projection value to a three-dimensional filter kernel to obtain a filtering result comprises: acquiring a weight of a neighborhood pixel of the two-dimensional feature point; inputting the weight and the normal projection value to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel at least comprises a two-dimensional filter kernel, and the two-dimensional filter kernel is a pixel value of the neighborhood pixel.

3. The method of claim 1, wherein, The acquiring three-dimensional space information of a two-dimensional feature point comprises: performing visual reconstruction on the two-dimensional feature point to obtain a three-dimensional coordinate corresponding to the two-dimensional feature point; the three-dimensional coordinate is the three-dimensional space information of the two-dimensional feature point.

4. The method of claim 3, wherein, The performing visual reconstruction on the two-dimensional feature point to obtain a three-dimensional coordinate corresponding to the two-dimensional feature point comprises: acquiring a first two-dimensional coordinate of the two-dimensional feature point on the image; acquiring two adjacent images which are adjacent frames to the image; acquiring two second two-dimensional coordinates of two adjacent two-dimensional feature points on the two adjacent images; the two adjacent two-dimensional feature points are obtained by imaging a same three-dimensional feature point as the two-dimensional feature point; calculating the three-dimensional coordinate based on the first two-dimensional coordinate and the two second two-dimensional coordinates.

5. The method of claim 4, wherein, The calculating the three-dimensional coordinate based on the first two-dimensional coordinate and the two second two-dimensional coordinates comprises: inputting the first two-dimensional coordinate and the two second two-dimensional coordinates to a coordinate conversion model to obtain a three-dimensional coordinate; the coordinate conversion model is constructed based on an imaging principle of a camera.

6. The method of claim 3, wherein, The determining a normal projection value of the two-dimensional feature point based on the three-dimensional space information comprises: projecting the three-dimensional feature point to the image to obtain the normal projection value of the two-dimensional feature point.

7. The method of claim 6, wherein, The projecting the three-dimensional feature point to the image to obtain the normal projection value of the two-dimensional feature point comprises: acquiring a normal of a face sheet corresponding to the three-dimensional feature point, and performing normalization processing on the normal; determining the normal projection value of the two-dimensional feature point based on an included angle between the normalized normal and a plane on which the image is located after projecting the three-dimensional feature point to the image.

8. A video stream filtering apparatus characterized by comprising: The device comprises: an acquisition module, configured to acquire three-dimensional space information of a two-dimensional feature point; the two-dimensional feature point is determined from a frame of image of a video stream; a determination module, configured to determine a normal projection value of the two-dimensional feature point based on the three-dimensional space information; a filtering module, configured to input the normal projection value to a three-dimensional filter kernel to obtain a filtering result; the three-dimensional filter kernel is used for filtering the video stream based on the normal projection value.

9. A video stream filtering device, characterized by The video stream filtering device comprises a memory, a processor and a video stream filtering program stored on the memory and executable on the processor, the video stream filtering program implementing the steps of the video stream filtering method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a video stream filtering program, the video stream filtering program implementing the steps of the video stream filtering method according to any one of claims 1 to 7 when executed by a processor.

Images