Shielding relation determination method and device, electronic equipment, medium and program product

Abstract

The invention provides an occlusion relation determination method and device, electronic equipment, a medium and a program product, and relates to the technical field of image processing. The method comprises the following steps: acquiring track information and optical flow information of each target object in at least two image frames; obtaining trajectory deviation information between the target objects according to the trajectory information of the target objects, and obtaining optical flow deviation information between the target objects according to the optical flow information of the target objects; obtaining a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects; and determining a shielding relationship between the target object and the virtual object according to a comparison relationship between the first depth and the second depth. According to the scheme provided by the embodiment of the invention, the virtual object generated by the special effect can be rendered according to the real shielding relationship between the real object and the virtual object in the video, and the problem that the shielding effect of the AR special effect is not real and a penetration phenomenon is easy to occur is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an occlusion relationship determination method, device, electronic equipment, medium and program product. Background Art

[0002] With the advancement of science and technology, augmented reality (AR) and virtual reality (VR) have become research hotspots. Taking AR as an example, it is a technology that increases users' perception of the real world through information enhanced by computer systems, and superimposes generated virtual objects, scenes or system prompt information on real scenes.

[0003] In the existing technical solutions, AR special effects are located on the top layer of all objects in the video screen, and cannot be integrated and interacted with other objects. Special effect objects are suspended on the top layer of the video screen, which will cause confusion in the three-dimensional relationship, lose the distance and occlusion relationship between the real object and the AR special effects, and cause the AR special effects to appear out of place. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, electronic device, medium and program product for determining an occlusion relationship, so as to solve the problem that the occlusion effect of AR special effects in the prior art is unrealistic and prone to mistakes.

[0005] In a first aspect, an embodiment of the present invention provides a method for determining an occlusion relationship, comprising:

[0006] Acquire trajectory information and optical flow information of each target object in at least two image frames, each of the image frames including at least two target objects;

[0007] According to the trajectory information of the target objects, trajectory deviation information between the target objects is obtained, and according to the optical flow information of the target objects, optical flow deviation information between the target objects is obtained;

[0008] Obtaining a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects;

[0009] The occlusion relationship between the target object and the virtual object is determined according to a comparison relationship between the first depth and a second depth, where the second depth is a depth of at least one virtual object generated based on an augmented reality special effect.

[0010] Further, the trajectory information includes point trajectory; the trajectory deviation information includes trajectory residual;

[0011] The obtaining the trajectory deviation information between the target objects according to the trajectory information of the target objects includes:

[0012] Calculating a basic matrix of a first target object according to the trajectory information of the target object, wherein the basic matrix is ​​used to characterize a constraint relationship between the first target object in different image frames;

[0013] Fitting the point trajectory of the second target object in the same image frame with the basic matrix to obtain the trajectory residual;

[0014] The first target object is any one of the target objects, and the second target object is all the target objects that are different from the first target object.

[0015] Further, the optical flow information includes an optical flow vector; the optical flow deviation information includes an optical flow motion residual;

[0016] The method of obtaining optical flow deviation information between the target objects according to the optical flow information of the target objects includes:

[0017] Obtaining an optical flow motion model of a first target object according to the optical flow information of the target object, wherein the optical flow motion model is used to characterize a motion trend of the first target object between different image frames;

[0018] Fitting the optical flow vector of the second target object in the same image frame with the optical flow motion model to obtain the optical flow motion residual;

[0019] The first target object is any one of the target objects, and the second target object is all the target objects that are different from the first target object.

[0020] Further, obtaining a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects includes:

[0021] A clustering result is obtained according to the trajectory deviation information and the optical flow deviation information, where the clustering result includes the first depth.

[0022] Further, obtaining a clustering result according to the trajectory deviation information and the optical flow deviation information includes:

[0023] constructing a first similarity matrix according to the trajectory deviation information, wherein the first similarity matrix is ​​used to characterize the trajectory deviation information between different image frames whose similarity is greater than a first preset value;

[0024] constructing a second similarity matrix according to the optical flow deviation information, wherein the second similarity matrix is ​​used to represent the optical flow deviation information between different image frames whose similarity is greater than a second preset value;

[0025] The first similarity matrix and the second similarity matrix are fused and clustered to obtain the clustering result.

[0026] Furthermore, the obtaining of the trajectory information and the optical flow information of each target object in at least two image frames includes:

[0027] The multiple target objects in the image frames are identified and tracked to obtain the trajectory information and the optical flow information of the multiple target objects in each frame of the image.

[0028] In a second aspect, an embodiment of the present invention provides an occlusion relationship determination device, comprising:

[0029] An acquisition module, used to acquire trajectory information and optical flow information of each target object in at least two image frames, each of the image frames including at least two target objects;

[0030] A first determination module is used to obtain trajectory deviation information between the target objects according to the trajectory information of the target objects, and to obtain optical flow deviation information between the target objects according to the optical flow information of the target objects;

[0031] A second determination module, configured to obtain a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects;

[0032] The third determination module is used to determine the occlusion relationship between the target object and the virtual object according to the comparison relationship between the first depth and the second depth, where the second depth is the depth of at least one virtual object generated based on the augmented reality special effect.

[0033] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the method for determining the occlusion relationship as described above is implemented.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a program or instruction stored thereon, wherein the program or instruction, when executed by a processor, implements the steps in the occlusion relationship determination method as described above.

[0035] In a fifth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the occlusion relationship determination method as described above.

[0036] The occlusion relationship determination method of the embodiment of the present invention can determine the trajectory deviation information and optical flow deviation information between the target objects through the trajectory information and optical flow information of each target object in at least two image frames, and then determine the first depth of the target object in the image frame, and finally determine the occlusion relationship between the target object and the virtual object by comparing the first depth with the second depth of the virtual object. The scheme of the embodiment of the present invention can render the virtual object generated by the special effect according to the real occlusion relationship between the real object and the virtual object in the video, which solves the problem that the occlusion effect of AR special effects is unrealistic and prone to goofs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a flow chart of a method for determining an occlusion relationship according to an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the occlusion relationship between the target object and the virtual object in an embodiment of the present invention;

[0039] Figure 3 This is a second schematic diagram of the occlusion relationship between the target object and the virtual object according to an embodiment of the present invention;

[0040] Figure 4 This is a third schematic diagram of the occlusion relationship between the target object and the virtual object according to an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of a method for determining an occlusion relationship according to an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of a module of an occlusion relationship determination device according to an embodiment of the present invention;

[0043] Figure 7 The figure is a schematic diagram of the implementation structure of the electronic device according to the embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present invention. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, for the sake of clarity and brevity, the description of the suppression function and structure is omitted.

[0045] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0046] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0047] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0048] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0049] like Figure 1 As shown, a method for determining an occlusion relationship according to an embodiment of the present invention includes the following steps:

[0050] Step 101: Acquire trajectory information and optical flow information of each target object in at least two image frames, each of the image frames including at least two target objects.

[0051] Optionally, the at least two image frames are continuous image frames of the video or image frames separated by a preset number of frames.

[0052] In the embodiment of the present invention, the number of the image frames is multiple, for example, all the image frames corresponding to the video; or, image frames with a preset number of frames in each interval among the image frames corresponding to the video.

[0053] It should be noted that, in the case of a video that requires higher precision and clarity, the image frames are continuous image frames, such as all image frames in a video;

[0054] When the accuracy and clarity of the video are not required to be high, the image frames are image frames with a preset number of frames per interval, for example, all image frames with a certain number of frames per interval in a video.

[0055] Optionally, obtaining trajectory information and optical flow information of each target object in at least two image frames includes:

[0056] The target object in the image frame is identified and tracked using a basic model to obtain the trajectory information and optical flow information of the target object.

[0057] It should be noted that the trajectory information is the change trend of the trajectory points of the target object in continuous or interval image frames;

[0058] Optical flow information refers to the motion pattern of pixels or objects in an image between consecutive or interval frames. It is used to reflect the motion information of objects in the video and can predict the motion state between two frames of images.

[0059] Step 102: obtaining trajectory deviation information between the target objects according to the trajectory information of the target objects, and obtaining optical flow deviation information between the target objects according to the optical flow information of the target objects.

[0060] Optionally, the trajectory deviation information between the target objects includes:

[0061] Track deviation information between any one of the target objects and all remaining target objects.

[0062] Optionally, the optical flow deviation information between the target objects includes:

[0063] Optical flow deviation information between any one of the target objects and all remaining target objects.

[0064] Step 103, obtaining a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects;

[0065] Step 104: determining an occlusion relationship between the target object and the virtual object based on a comparison relationship between the first depth and a second depth, wherein the second depth is a depth of at least one virtual object generated based on an augmented reality special effect.

[0066] Optionally, the occlusion relationship between the target object and the virtual object includes:

[0067] The occlusion relationship between the multiple target objects and at least one virtual object.

[0068] Determining the occlusion relationship between the target object and the virtual object according to the comparison relationship between the first depth and the second depth can be understood as:

[0069] In order of depth from small to large, the target object or the virtual object is occluded and superimposed in sequence from the top layer to the bottom layer.

[0070] In the embodiment of the present invention, the target object includes a first target object and a second target object, and the virtual object is one, then:

[0071] In the case where the depth comparison relationship is: the third target object>the fourth target object>the virtual object, such as Figure 2 As shown, the occlusion relationship is: the virtual object is at the top layer, the fourth target object is at the middle layer, and the third target object is at the bottom layer;

[0072] Specifically, there is no need to perform superimposition processing on the target object in the image frame, and it is only necessary to render the virtual object on the top layer.

[0073] In the embodiment of the present invention, the target object includes a third target object and a fourth target object, and the virtual object is one, then:

[0074] In the case where the depth comparison relationship is: the third target object>the virtual object>the fourth target object, such as Figure 3 As shown, the occlusion relationship is: the fourth target object is at the top layer, the virtual object is at the middle layer, and the third target object is at the bottom layer;

[0075] Specifically, there is no need to perform superimposition processing on the third target object, and only the virtual object and the fourth target object are superimposed, the fourth target object is displayed on the top layer, the virtual object is displayed on the middle layer, and the third target object is displayed on the bottom layer.

[0076] In the embodiment of the present invention, the target object includes a third target object and a fourth target object, and the virtual object is one, then:

[0077] In the case where the depth comparison relationship is: virtual object>fourth target object>third target object, if Figure 4 As shown, the occlusion relationship is: the third target object is at the top layer, the fourth target object is at the middle layer, and the virtual object is at the bottom layer;

[0078] Specifically, after rendering the virtual object, the virtual object is superimposed on the third target object and then on the fourth target object, the third target object is displayed on the top layer, the fourth target object is displayed on the middle layer, and the virtual object is displayed on the bottom layer.

[0079] The occlusion relationship determination method of the embodiment of the present invention can determine the trajectory deviation information and optical flow deviation information between the target objects through the trajectory information and optical flow information of each target object in at least two image frames, and then determine the first depth of the target object in the image frame, and finally determine the occlusion relationship between the target object and the virtual object by comparing the first depth with the second depth of the virtual object. The scheme of the embodiment of the present invention can render the virtual object generated by the special effect according to the real occlusion relationship between the real object and the virtual object in the video, which solves the problem that the occlusion effect of AR special effects is unrealistic and prone to goofs.

[0080] Optionally, acquiring the trajectory information and optical flow information of each target object in at least two image frames includes:

[0081] The target object is identified and tracked using a basic model to obtain a point trajectory and an optical flow mask of each target object in at least two image frames.

[0082] It should be noted that the point trajectory is a sparse point trajectory; and the optical flow mask is a dense optical flow.

[0083] Optionally, the method further comprises:

[0084] An optical flow vector is determined according to the optical flow mask and a preset optical flow vector in each image frame.

[0085] Optionally, the trajectory information includes a point trajectory; the trajectory deviation information includes a trajectory residual;

[0086] The obtaining the trajectory deviation information between the target objects according to the trajectory information of the target objects includes:

[0087] Calculating a basic matrix of a first target object according to the trajectory information of the target object, wherein the basic matrix is ​​used to characterize a constraint relationship between the first target object in different image frames;

[0088] Fitting the point trajectory of the second target object in the same image frame with the basic matrix to obtain the trajectory residual;

[0089] The first target object is any one of the target objects, and the second target object is all the target objects that are different from the first target object.

[0090] Optionally, calculating a basic matrix of the first target object according to the trajectory information of the target object includes:

[0091] An eight-point algorithm is used to solve the point trajectory of the first target object to obtain a basic matrix of the first target object between every N image frames;

[0092] Wherein, N is an integer greater than or equal to 1.

[0093] It should be noted that the basic matrix is ​​a motion model based on epipolar geometry.

[0094] Optionally, fitting the point trajectory of the second target object in the same image frame with the basic matrix to obtain the trajectory residual includes:

[0095] Based on the Sampson error, the basic matrix of the same image frame is fitted to the point trajectory of the second target object to obtain the trajectory residual.

[0096] It should be noted that the Sampson error is an error whose complexity lies between the algebraic error and the geometric error, but is very close to the geometric error;

[0097] The trajectory residual is the mean square error.

[0098] In the embodiment of the present invention, the number of the target objects is k. For the i-th target object in the m-th image frame, the trajectory residual can be expressed as:

[0099] Wherein, k is greater than or equal to 2, m is greater than 1, and i is greater than 0; is the mean residual error (Sampson error) used to fit the fundamental matrix of target object i to the trajectory of target object k between frames m and m+f.

[0100] The occlusion relationship determination method of the embodiment of the present invention can provide high-precision data support for the segmentation of target objects in a video or image by calculating the trajectory residuals between the target objects, so that the segmented target objects are more accurate.

[0101] Optionally, the optical flow information includes an optical flow vector; the optical flow deviation information includes an optical flow motion residual;

[0102] The step of obtaining optical flow deviation information between the target objects according to the optical flow information of the target objects includes:

[0103] Obtaining an optical flow motion model of a first target object according to the optical flow information of the target object, wherein the optical flow motion model is used to characterize a motion trend of the first target object between different image frames;

[0104] Fitting the optical flow vector of the second target object in the same image frame with the optical flow motion model to obtain the optical flow motion residual;

[0105] The first target object is any one of the target objects, and the second target object is all the target objects that are different from the first target object.

[0106] Optionally, obtaining an optical flow motion model of a first target object according to the optical flow information of the target object includes:

[0107] The full secondary motion model is fitted to the optical flow vector of the second target object in the optical flow information to obtain the optical flow motion model.

[0108] It should be noted that the optical flow motion model is determined by calculating the displacement vector between adjacent frames, and the optical flow motion model can estimate the displacement or motion trend of an object in an image.

[0109] In the embodiment of the present invention, the full quadratic motion model is a full quadratic motion model including 12 parameters; this motion model is usually used to simulate the instantaneous motion of a screw of an object, and simulates the instantaneous spiral motion of a rigid object at any depth;

[0110] The expression of the full quadratic motion model including 12 parameters is:

[0111] f(x,y)=(a+bx+cy+dx 2 +exy+fy 2 ,g+hx+iy+jx 2 +kxy+ly 2 );

[0112] Wherein, (x, y) is the pixel or object (the target object) relative to the center of the 2D coordinate image, and a, b, c...k, l are the 12 parameters;

[0113] Let f(x,y)=(u,v);

[0114] Among them, (u, v) is the optical flow vector of the target object, and a full quadratic motion model containing 12 parameters is fitted to the optical flow vector of each of the second target objects. The 12 parameters representing the object are solved by optimizing the mean square error to determine the optical flow motion model.

[0115] In real application scenarios, when the shooting device is rotated or displaced significantly, slight deviations in position are likely to occur before repositioning, resulting in different deviations in experience at different angles.

[0116] In the embodiment of the present invention, a full quadratic motion model is used to correct the deviation caused by the rotation of the shooting device.

[0117] Optionally, fitting the optical flow vector of the second target object in the same image frame with the optical flow motion model to obtain the optical flow motion residual includes:

[0118] Based on the Sampson error, the optical flow motion model of the same image frame is fitted to the optical flow vector of the second target object to obtain the optical flow motion residual.

[0119] In the embodiment of the present invention, the number of the target objects is k. For the i-th target object in the m-th image frame, the optical flow motion residual can be expressed as:

[0120] Wherein, k is greater than or equal to 2, m is greater than 1, and i is greater than 0; is the average residual (Sampson error) of fitting the optical flow motion model of target object i to the optical flow vector of target object k between image frames m and m+1.

[0121] The occlusion relationship determination method of the embodiment of the present invention can provide high-precision data support for the segmentation of target objects in videos or images by calculating the optical flow motion residuals between the target objects, so that the segmented target objects are more accurate.

[0122] Optionally, obtaining a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects includes:

[0123] A clustering result is obtained according to the trajectory deviation information and the optical flow deviation information, where the clustering result includes the first depth.

[0124] Optionally, obtaining a clustering result according to the trajectory deviation information and the optical flow deviation information includes:

[0125] Using a clustering algorithm to obtain a clustering result according to the trajectory deviation information and the optical flow deviation information;

[0126] Wherein, the clustering algorithm is a common regularization multi-view spectral clustering algorithm.

[0127] It should be noted that co-regularization is an effective technique in semi-supervised multi-view learning. The core idea of ​​co-regularization is to minimize the difference between the prediction functions of the two views as part of the objective function. That is, the difference between the trajectory deviation information and the optical flow deviation information is minimized, and the trajectory deviation information and the optical flow deviation information are clustered.

[0128] Optionally, obtaining the clustering result according to the trajectory deviation information and the optical flow deviation information includes:

[0129] constructing a first similarity matrix according to the trajectory deviation information, wherein the first similarity matrix is ​​used to characterize the trajectory deviation information between different image frames whose similarity is greater than a first preset value;

[0130] constructing a second similarity matrix according to the optical flow deviation information, wherein the second similarity matrix is ​​used to represent the optical flow deviation information between different image frames whose similarity is greater than a second preset value;

[0131] The first similarity matrix and the second similarity matrix are fused and clustered to obtain the clustering result.

[0132] Optionally, constructing a first similarity matrix according to the trajectory deviation information includes:

[0133] Using an ordered residual kernel algorithm, constructing the first similarity matrix according to the trajectory residuals;

[0134] Constructing a second similarity matrix according to the optical flow deviation information includes:

[0135] The second similarity matrix is ​​constructed according to the optical flow motion residual using an ordered residual kernel algorithm.

[0136] Optionally, performing fusion clustering on the first similarity matrix and the second similarity matrix to obtain the first depth includes:

[0137] A common regularized multi-view spectral clustering algorithm is used to perform fusion clustering on the first similarity matrix and the second similarity matrix to obtain a clustering result including the first depth.

[0138] The occlusion relationship determination method of the embodiment of the present invention can segment multiple target objects in the video by fusing and clustering the trajectory deviation information and the optical flow deviation information, thereby improving the accuracy of image segmentation.

[0139] Optionally, the method further comprises:

[0140] Performing cloud positioning on each frame of the image to obtain the position information of the shooting device that shot the image frame;

[0141] A second depth of at least one of the virtual objects is determined according to the posture information of the shooting device.

[0142] In an embodiment of the present invention, the shooting device is a camera, and in the process of acquiring the trajectory information and optical flow information of each target object in at least two image frames, the at least two image frames are positioned in the cloud to determine the position information of the camera;

[0143] According to the shown posture information, the depth and posture of the virtual object relative to the target object in each image frame are determined.

[0144] The occlusion relationship determination method of an embodiment of the present invention determines the second depth of the virtual object through cloud positioning, and determines the real occlusion relationship between the target object and the virtual object by comparing the second depth with the first depth, thereby rendering and displaying the target object and the virtual object in the video according to the real occlusion relationship.

[0145] like Figure 5 As shown, the occlusion relationship determination method of the embodiment of the present invention includes:

[0146] Obtain image frame data in the video, perform cloud positioning on each frame, and determine the position and posture of the shooting device;

[0147] Identify, detect and track the target object in the image frame to obtain a point trajectory and an optical flow mask of the target object;

[0148] Fitting and clustering the point trajectory and the optical flow mask respectively to obtain a clustering result;

[0149] Determine first depth information of the target object according to the clustering result, and determine second depth information of the virtual object according to the position and posture of the shooting device;

[0150] The first depth information is compared with the second depth information one by one to determine the occlusion relationship between the target object and the virtual object.

[0151] like Figure 6 As shown, an embodiment of the present invention provides an occlusion relationship determination device 600, comprising:

[0152] An acquisition module 601 is used to acquire trajectory information and optical flow information of each target object in at least two image frames, each of the image frames including at least two target objects;

[0153] A first determination module 602 is used to obtain trajectory deviation information between the target objects according to the trajectory information of the target objects, and obtain optical flow deviation information between the target objects according to the optical flow information of the target objects;

[0154] A second determination module 603 is used to obtain a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects;

[0155] The third determination module 604 is used to determine the occlusion relationship between the target object and the virtual object according to the comparison relationship between the first depth and the second depth, where the second depth is the depth of at least one virtual object generated based on the augmented reality special effect.

[0156] The occlusion relationship determination device of the embodiment of the present invention can determine the trajectory deviation information and optical flow deviation information between the target objects through the trajectory information and optical flow information of each target object in at least two image frames, and then determine the first depth of the target object in the image frame, and finally determine the occlusion relationship between the target object and the virtual object by comparing the first depth with the second depth of the virtual object. The scheme of the embodiment of the present invention can render the virtual object generated by the special effect according to the real occlusion relationship between the real object and the virtual object in the video, which solves the problem that the occlusion effect of AR special effects is unrealistic and prone to goofs.

[0157] like Figure 7 As shown, an embodiment of the present invention provides an electronic device, including a transceiver 710, a processor 700, a memory 720, and a program or instruction stored on the memory 720 and executable on the processor 700; when the processor 700 executes the program or instruction, the above-mentioned occlusion relationship determination method applied as described above is implemented.

[0158] The transceiver 710 is used to receive and send data under the control of the processor 700.

[0159] Among them, Figure 7 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 700 and various circuits of memory represented by memory 720 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 710 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 730 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0160] The processor 700 is responsible for managing the bus architecture and general processing, and the memory 720 can store data used by the processor 700 when performing operations.

[0161] Optionally, the transceiver 710 is used to obtain trajectory information and optical flow information of each target object in at least two image frames, and each of the image frames includes at least two target objects;

[0162] The processor 700 is used to obtain trajectory deviation information between the target objects according to the trajectory information of the target objects, and obtain optical flow deviation information between the target objects according to the optical flow information of the target objects;

[0163] Obtaining a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects;

[0164] The occlusion relationship between the target object and the virtual object is determined according to a comparison relationship between the first depth and a second depth, where the second depth is a depth of at least one virtual object generated based on an augmented reality special effect.

[0165] An embodiment of the present invention provides a computer-readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the above-mentioned occlusion relationship determination method are implemented.

[0166] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0167] An embodiment of the present invention provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0168] It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0169] In the embodiment of the present invention, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0170] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0171] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0172] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present invention. Therefore, the present invention should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present invention will be perfect and complete, and the scope of the present invention will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "including" and / or "comprising" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0173] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for determining an occlusion relationship, characterized in that: include: Acquire trajectory information and optical flow information of each target object in at least two image frames, each of the image frames including at least two target objects; According to the trajectory information of the target objects, trajectory deviation information between the target objects is obtained, and according to the optical flow information of the target objects, optical flow deviation information between the target objects is obtained; Obtaining a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects; The occlusion relationship between the target object and the virtual object is determined according to a comparison relationship between the first depth and a second depth, where the second depth is a depth of at least one virtual object generated based on an augmented reality special effect.

2. The method for determining occlusion relationship according to claim 1, characterized in that: The trajectory information includes point trajectory; the trajectory deviation information includes trajectory residual; The obtaining the trajectory deviation information between the target objects according to the trajectory information of the target objects includes: Acquire a basic matrix of a first target object according to the trajectory information of the target object, wherein the basic matrix is ​​used to characterize a constraint relationship between the first target object in different image frames; Fitting the point trajectory of the second target object in the same image frame with the basic matrix to obtain the trajectory residual; The first target object is any one of the target objects, and the second target object is all the target objects that are different from the first target object.

3. The method for determining occlusion relationship according to claim 1, characterized in that: The optical flow information includes an optical flow vector; the optical flow deviation information includes an optical flow motion residual; The step of obtaining optical flow deviation information between the target objects according to the optical flow information of the target objects includes: Obtaining an optical flow motion model of a first target object according to the optical flow information of the target object, wherein the optical flow motion model is used to characterize a motion trend of the first target object between different image frames; Fitting the optical flow vector of the second target object in the same image frame with the optical flow motion model to obtain the optical flow motion residual; The first target object is any one of the target objects, and the second target object is all the target objects that are different from the first target object.

4. The method for determining occlusion relationship according to claim 1, characterized in that: The obtaining, according to the trajectory deviation information and the optical flow deviation information between the target objects, a first depth of the target object in the image frame comprises: A clustering result is obtained according to the trajectory deviation information and the optical flow deviation information, where the clustering result includes the first depth.

5. The method for determining occlusion relationship according to claim 4, characterized in that: The obtaining a clustering result according to the trajectory deviation information and the optical flow deviation information includes: constructing a first similarity matrix according to the trajectory deviation information, wherein the first similarity matrix is ​​used to characterize the trajectory deviation information between different image frames whose similarity is greater than a first preset value; constructing a second similarity matrix according to the optical flow deviation information, wherein the second similarity matrix is ​​used to represent the optical flow deviation information between different image frames whose similarity is greater than a second preset value; The first similarity matrix and the second similarity matrix are fused and clustered to obtain the clustering result.

6. The method for determining occlusion relationship according to claim 1, characterized in that: The step of obtaining the trajectory information and the optical flow information of each target object in at least two image frames includes: The multiple target objects in the image frames are identified and tracked to obtain the trajectory information and the optical flow information of the multiple target objects in each frame of the image.

7. A device for determining an occlusion relationship, characterized in that: include: An acquisition module, used to acquire trajectory information and optical flow information of each target object in at least two image frames, each of the image frames including at least two target objects; A first determination module is used to obtain trajectory deviation information between the target objects according to the trajectory information of the target objects, and to obtain optical flow deviation information between the target objects according to the optical flow information of the target objects; A second determination module, configured to obtain a first depth of the target object in the image frame according to the trajectory deviation information and the optical flow deviation information between the target objects; The third determination module is used to determine the occlusion relationship between the target object and the virtual object according to the comparison relationship between the first depth and the second depth, where the second depth is the depth of at least one virtual object generated based on the augmented reality special effect.

8. An electronic device, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; characterized in that when the processor executes the program or instruction, the method for determining an occlusion relationship as described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps in the occlusion relationship determination method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method for determining an occlusion relationship as claimed in any one of claims 1 to 6.

Images