Method for stereoscopic tagging of videos

Abstract

The application provides a method for adding a stereoscopic label to a video, characterized by comprising the following steps: extracting three-dimensional coordinate system information of a video shot by a camera; determining three-dimensional coordinates of a label according to a selection point of a user on a screen; and synthesizing the video shot by the camera and the label. The method for adding a dynamic stereoscopic label to a video can realize dynamic identification of objects in the video with a sense of depth and a stereoscopic effect, so as to facilitate a video viewer to quickly capture information of interest, and has high commercial value in the fields of advertisement, guiding, teaching and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to a video processing technology, in particular to a method for adding dynamic 3D markers in a video. BACKGROUND

[0002] With the development of computer technology and network technology, the network speed is getting faster and faster, and the processing capability of the terminal is getting stronger and stronger. Video-based applications are becoming more and more popular. On the other hand, the information carrying capacity of a video is large, and people's life rhythm is getting faster and faster. Therefore, if different transactions in a video can be marked, it is beneficial to prompt the viewer, assist the viewer to quickly filter information, and deepen the impression. There is such a demand in the scenarios of advertising, teaching, and tourist attraction introduction.

[0003] Currently, the main way to mark a video is to rely on a video post-editing personnel to complete by adding a 2D text label. The disadvantages of this approach are obvious: the 2D text label itself cannot change with the object it identifies, and when the label is relatively dense, it can easily cause stacking and affect recognition. Moreover, when the observer (camera) position and angle change, the 2D label cannot change accordingly with the object it identifies, resulting in a poor viewing experience.

[0004] Therefore, if a shooting device can be used to directly provide a 3D label adding function when shooting a video, a large amount of video post-editing work can be avoided, and the visual experience of the label can be improved, which becomes a problem to be solved. SUMMARY

[0005] In view of the above-mentioned disadvantages of the prior art, the purpose of the present application is to provide a method for adding a stereoscopic label to a video, characterized by comprising: extracting three-dimensional coordinate system information of a video shot by a camera; determining three-dimensional coordinates of a label according to a selection point of a user on a screen; and synthesizing the video shot by the camera and the label.

[0006] Preferably, in the method for adding a stereoscopic label to a video, the step of extracting three-dimensional coordinate system information of a video shot by a camera comprises the step of positioning the camera using a visual inertial system and an inertial navigation system.

[0007] Preferably, in the method for adding a stereoscopic label to a video, the step of determining three-dimensional coordinates of a label comprises: setting initial three-dimensional coordinates of the label according to screen horizontal and vertical coordinates of the selection point and a default initial Z-axis coordinate; changing the initial three-dimensional coordinates according to a position change of the camera relative to the three-dimensional coordinates; and confirming the three-dimensional coordinates of the label according to a confirmation instruction of the user.

[0008] Preferably, in the method for adding a stereoscopic label to a video, the step of determining the three-dimensional coordinates of the label comprises: setting initial three-dimensional coordinates of the label according to screen horizontal and vertical coordinates of the selected point and a default initial Z-axis coordinate; changing the Z-axis coordinate of the label according to user input; and confirming the three-dimensional coordinates of the label according to a confirmation instruction of the user.

[0009] Preferably, in the method for adding a stereoscopic label to a video, the step of changing the Z-axis coordinate of the label according to user input comprises: increasing the Z-axis of the label by Δz per millisecond when the user input is long pressing the label; and decreasing the Z-axis coordinate of the label by Δz per double-click when the user input is double-clicking the label.

[0010] Preferably, in the method for adding a stereoscopic label to a video, the step of changing the Z-axis coordinate of the label according to user input comprises: increasing the Z-axis of the label by Δz per millisecond when the user input is long pressing the label; and decreasing the Z-axis coordinate of the label by Δz per double-click when the user input is double-clicking the label.

[0011] Preferably, in the method for adding a stereoscopic label to a video, the step of determining the three-dimensional coordinates of the label comprises: identifying an object on the selected point and setting three-dimensional coordinates of the object as initial three-dimensional coordinates of the label.

[0012] Preferably, in the method for adding a stereoscopic label to a video, the step of synthesizing the video captured by the camera with the label further comprises the steps of visual calibration and inertial calibration.

[0013] The present application is a method for adding a dynamic stereoscopic label to a video, which can realize dynamic identification of objects in a video with a sense of depth and a stereoscopic effect, thereby facilitating a video viewer to quickly capture information of interest and having high commercial value in the fields of advertising, guiding, teaching, etc. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The figure shows a flowchart of an embodiment of the present application;

[0015] Figure 2 The figure shows Figure 1 The figure shows a schematic diagram of a lens local coordinate system used in the embodiment. DETAILED DESCRIPTION

[0016] Following, the embodiments of the present application are described by specific examples, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the specification. The present application can also be implemented or applied by other different embodiments, and various modifications or changes can be made to the details in the specification based on different views and applications without departing from the spirit of the present application.

[0017] Figure 1 The figure shows a flowchart of an embodiment of the present application. Step S1 represents extracting three-dimensional coordinate system information of the video taken by the camera. To realize the depth and three-dimensional effect of the mark, the spatial coordinates (three-dimensional coordinates) of the marked object in the coordinate system should be determined. However, the coordinate system is relative, and the three-dimensional coordinate system can have a world coordinate (absolute coordinate) system and a local coordinate system. The video is usually formed by recording the real world with a camera device, so the actual video embodies the relative position and direction between the object and the lens with the lens as the origin, that is, the video content adopts a local coordinate system equivalent to the lens.

[0018] The recognition of the coordinate system must solve the positioning problem. The positioning of video shooting is closer to the indoor positioning problem, and cannot use GPS, Beidou and other navigation systems to realize the positioning in the world coordinate system. The positioning problem of video shooting solves the 6 degrees of freedom of the camera itself (referring to the movement in xyz three directions plus pitch / deflection / roll), as well as the relative position relationship between the object and the environment. The way to solve the positioning problem of video shooting is to comprehensively use visual inertial system and inertial navigation system.

[0019] The visual inertial system uses the continuous frames taken by the camera as input, recognizes the feature points in each frame of image, and identifies the position and posture of the camera according to the position change of the feature points between the continuous image frames. The inertial navigation system uses the continuous readings of inertial measurement units such as accelerometers and gyroscopes to calculate the posture and position change of the camera. Combining the calculation results of the two systems (for example, using Kalman filter), the accuracy of camera positioning can be optimized. It is assumed that the local coordinate system of the lens in this example is as shown in Figure 2 , wherein the origin C represents the camera, and the plane S represents a plane perpendicular to the Z axis and parallel to the screen. Unless otherwise specified, the three-dimensional coordinates also refer to the coordinates shown in Figure 2 . There are some tools that can complete similar work at present, such as ARKit tool of Apple Inc., ARCore of Google Inc., etc.

[0020] Step S2 represents determining the three-dimensional coordinates of the label. Taking the case of shooting a video scene with a mobile phone as an example, the position of the 3D label is determined by the user clicking the screen of the mobile phone. However, the screen is a two-dimensional system, so the user's click can only obtain a two-dimensional coordinate (x1, y1) in a screen coordinate system. After converting the two-dimensional coordinate (x1, y1) into three-dimensional coordinates, the three-dimensional coordinates can be used as the position of the 3D label. The coordinate (x1, y1) and the horizontal and vertical coordinates in the lens coordinate system can be considered as a one-to-one correspondence, so the conversion is relatively simple and will not be described in detail. The main problem is the determination of the Z-axis coordinate. Usually, the Z-axis coordinate of the label position can be achieved by receiving the user's adjustment instruction and confirmation instruction. There are several implementation methods as follows:

[0021] Scheme one: first set a default Z-axis coordinate z1, and use (x1, y1, z1) as the initial three-dimensional coordinates of the label; at the same time, set the label as a stationary object at the three-dimensional coordinates. In the second step, the user adjusts the relative position of the camera with respect to the "stationary" label by moving the camera, and at this time the three-dimensional coordinates of the label will change. In the operation, the visual-inertial system and inertial navigation system can be used. In the third step, after the user adjusts the relative position of the camera with respect to the "stationary" label, the three-dimensional coordinates at this time are received as the final three-dimensional coordinates of the label.

[0022] Scheme two: first set a default Z-axis coordinate z1, and use (x1, y1, z1) as the initial three-dimensional coordinates of the label; in the second step, listen to the user input. The user long-presses the label to increase the z1 coordinate of the label, and double-clicks to decrease the z1 coordinate of the label. For example, when the user's long-press is detected, the z1 coordinate of the label is increased by Δz per millisecond; when the user's double-click on the label is detected, or when the user's continuous clicks on the label are detected, the z1 coordinate of the label is decreased by Δz per double-click, or the z1 coordinate of the label is decreased by Δz per continuous click. In the third step, the three-dimensional coordinates at this time are received as the final three-dimensional coordinates of the label.

[0023] Scheme three: first identify whether there is an object at the user's selection point. If the identification is successful, the three-dimensional coordinates of the object are directly used as the initial three-dimensional coordinates of the label (to prevent the label from overlapping with the object, the coordinates can be increased or decreased appropriately by default), and then the three-dimensional coordinates at this time are received as the final three-dimensional coordinates of the label after receiving the user's confirmation instruction. The object recognition algorithm has been relatively successful, and the recognition model can be trained through machine exercises. This is not the point of the present invention and will not be described in detail.

[0024] In addition, the three schemes can be used simultaneously, for example, the combination of scheme one and scheme two: the user moves the camera away from the tag while long pressing the tag, and the Z-axis coordinate of the tag increases at a faster rate. Scheme two and scheme three can also be combined: the user can adjust the tag position determined by the system after recognizing the object by long pressing, double clicking, and the like. The combination of different schemes is set by those skilled in the art according to actual needs, and does not exceed the scope of the idea of the present application.

[0025] Step S3 represents the synthesis of the video taken by the camera and the tag. After obtaining the three-dimensional information of the tag, the real scene photographed by the camera can be synthesized with the tag to obtain a video with 3D tag information. However, the points on the imaging system sensor and the points in the real world do not necessarily match completely. In order to improve the synthesis quality, visual calibration (including geometric calibration and optical calibration), inertial calibration, and transformation according to the intrinsic and extrinsic parameters of the camera are often required before synthesis.

[0026] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. For example, steps S2 and S3 have no sequence, and the tag can be rendered first, and then the coordinates of the tag are determined, or the coordinates can be determined first, and then the tag is rendered. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed by the present application should be covered by the claims of the present application.

Claims

1. A method of adding a stereoscopic tag to a video, characterized by, The application relates to a method for synthesizing a video and a label, comprising the following steps: extracting three-dimensional coordinate system information of a video taken by a camera, wherein the three-dimensional coordinate system information is three-dimensional coordinate system information in a local coordinate system of the camera; determining three-dimensional coordinates of a label according to a selection point of a user on a screen, comprising the following steps: setting initial three-dimensional coordinates of the label according to screen horizontal and vertical coordinates of the selection point and a default initial Z-axis coordinate; adjusting a position of the camera by the user moving the camera, changing the initial three-dimensional coordinates according to a position change of the camera relative to the three-dimensional coordinates; confirming the three-dimensional coordinates of the label according to a confirmation instruction of the user; synthesizing the video taken by the camera and the label.

2. The method of stereoscopic tagging of video of claim 1, wherein, In the step of extracting three-dimensional coordinate system information of a video taken by a camera, the step of positioning the camera by using a visual inertial system and an inertial navigation system is included.

3. The method of stereoscopic tagging of video of claim 1, wherein, In the step of determining three-dimensional coordinates of a label, the step of identifying an object on the selection point and taking three-dimensional coordinates of the object as initial three-dimensional coordinates of the label is included.

4. The method of stereotagging video of claim 1, wherein, The step of synthesizing the video taken by the camera and the label further includes the steps of visual calibration and inertial calibration.

Images