Motion track generation method and device and computer equipment

Abstract

The invention relates to a motion trail generation method and device and computer equipment. The method comprises the following steps: acquiring a binocular video stream; the binocular video stream is a video stream synchronously acquired by a left lens and a right lens in the integrated binocular camera; according to the binocular video stream, scene real-time depth information corresponding to the binocular video stream is obtained; and generating a motion track of the target object in the binocular video stream based on the binocular video stream and the scene real-time depth information corresponding to the binocular video stream. By adopting the method, the problem that the parabolic point cannot be accurately positioned due to the inaccurate trajectory path of the parabolic body can be solved.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a motion trajectory generation method, device and computer equipment. Background Art

[0002] At present, the real-time video stream of high-altitude objects collected by a single or multiple cameras is generally used to detect the trajectory of high-altitude objects. Traditional detection methods often have poor clarity for monitoring images of higher floors or longer distances, resulting in inaccurate trajectory paths of objects recorded in the video stream, so it is impossible to extend the location to the point of throwing, which is of limited help in subsequent evidence collection and accountability.

[0003] However, for the existing motion trajectory generation method, there is still no effective solution to the problem that the parabolic point cannot be accurately located due to the inaccurate trajectory path of the parabolic object. Summary of the invention

[0004] Based on this, it is necessary to provide a motion trajectory generation method, device and computer equipment to address the above technical problems.

[0005] In a first aspect, the present application provides a motion trajectory generation method. The method comprises the following steps:

[0006] Obtain a binocular video stream; the binocular video stream is a video stream synchronously collected by left and right lenses of an integrated binocular camera;

[0007] According to the binocular video stream, real-time depth information of the scene corresponding to the binocular video stream is obtained;

[0008] Based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, a motion trajectory of the target object in the binocular video stream is generated.

[0009] In one of the embodiments, generating a motion trajectory of a target object in the binocular video stream based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream comprises the following steps:

[0010] Based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, determining the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene;

[0011] Based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene, a motion trajectory of the target object in the binocular video stream is generated.

[0012] In one embodiment, generating the motion trajectory of the target object in the binocular video stream based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene comprises the following steps:

[0013] Determine motion trajectory estimation information of the target object based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene;

[0014] Based on the motion trajectory estimation information of the target object, dynamically adjusting the region of interest in the binocular video stream corresponding to the motion trajectory estimation information of the target object;

[0015] Based on the adjusted region of interest, a motion trajectory of the target object in the binocular video stream is generated.

[0016] In one embodiment, determining the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream comprises the following steps:

[0017] Based on the binocular video stream, determining the position of the target object in each image frame in the binocular video stream;

[0018] Determine the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream;

[0019] Based on the real-time position, the real-time movement direction and the real-time movement speed of the target object in the binocular video stream, and the real-time depth information of the scene corresponding to the binocular video stream, the real-time position information and the real-time movement information of the target object in the binocular video stream in the three-dimensional scene are determined.

[0020] In one of the embodiments, determining the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream comprises the following steps:

[0021] Based on the position of the target object in each image frame in the binocular video stream, determining a positional relationship between the position of the target object in the current image frame in the binocular video stream and the positions of the target objects in the previous and next image frames;

[0022] Based on the positional relationship between the position of the target object in the current image frame in the binocular video stream and the position of the target object in the previous and next image frames, the real-time position, the real-time movement direction and the real-time movement speed of the target object in the binocular video stream are determined.

[0023] In one embodiment, obtaining the real-time depth information of the scene corresponding to the binocular video stream according to the binocular video stream comprises the following steps:

[0024] Determine, according to the binocular video stream, a real-time binocular parallax relationship between left and right lenses of the integrated binocular camera corresponding to the binocular video stream;

[0025] Based on the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream, the real-time depth information of the scene corresponding to the binocular video stream is obtained.

[0026] In one embodiment, determining the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream according to the binocular video stream comprises the following steps:

[0027] Based on the binocular video stream, determining each pair of image frames of the binocular video stream; the image frame pair is a pair of image frames synchronously captured by left and right lenses in the integrated binocular camera;

[0028] Based on each of the image frame pairs, a real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream is determined.

[0029] In a second aspect, the present application also provides a motion trajectory generation device. The device comprises:

[0030] An acquisition module, used for synchronously acquiring video streams using left and right lenses of an integrated binocular camera;

[0031] A calculation module is used to determine the position of a target object in each image frame in the binocular video stream based on the binocular video stream; determine the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream; calculate the real-time position information and real-time movement information of the target object in the binocular video stream in a three-dimensional scene based on the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream;

[0032] And an encoding module, which is used to determine the motion trajectory prediction information of the target object based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene; based on the motion trajectory prediction information of the target object, dynamically adjust the area of ​​interest in the binocular video stream corresponding to the motion trajectory prediction information of the target object.

[0033] In a third aspect, the present application also provides a motion trajectory generation system. The system comprises:

[0034] An acquisition module is used to acquire a binocular video stream; the binocular video stream is a video stream synchronously collected by the left and right lenses of the integrated binocular camera;

[0035] A depth information determination module, used to obtain real-time depth information of the scene corresponding to the binocular video stream according to the binocular video stream;

[0036] And a trajectory generation module is used to generate a motion trajectory of a target object in the binocular video stream based on the binocular video stream and real-time depth information of the scene corresponding to the binocular video stream.

[0037] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the motion trajectory generation method described in the first aspect is implemented.

[0038] The above-mentioned motion trajectory generation method, device and computer equipment obtain the real-time depth information of the scene corresponding to the binocular video stream through the binocular video stream, and then accurately judge the motion trajectory of the target object in the three-dimensional scene according to the real-time binocular video stream and the real-time depth information of the scene, and obtain the motion trajectory of the target object in the binocular video stream. By obtaining the accurate motion trajectory, the parabola point can be accurately located according to the clear motion trajectory, which solves the problem that the parabola point cannot be accurately located due to the inaccurate trajectory path of the parabola.

[0039] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 A hardware structure block diagram of a terminal of a motion trajectory generation method provided in an embodiment of the present application;

[0042] Figure 2 A flow chart of a motion trajectory generation method provided in one embodiment of the present application;

[0043] Figure 3 A flow chart of a motion trajectory generation method provided in a preferred embodiment of the present application;

[0044] Figure 4 A structural block diagram of a motion trajectory generating device provided in an embodiment of the present application;

[0045] Figure 5 This is a structural block diagram of a motion trajectory generation system provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0047] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0048] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the motion trajectory generation method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0049] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the motion trajectory generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0051] In this embodiment, a motion trajectory generation method is provided. Figure 2 is a flow chart of the motion trajectory generation method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0052] Step S210, obtaining a binocular video stream; the binocular video stream is a video stream synchronously captured by the left and right lenses of an integrated binocular camera.

[0053] The above-mentioned binocular video stream is a video stream synchronously collected by the left and right lenses in the integrated binocular camera, and specifically can be a video stream synchronously collected by a monitoring device. The above-mentioned monitoring device can be a video acquisition device equipped with an integrated binocular camera. The above-mentioned integrated binocular camera can include a pre-integrated left camera and a right camera, each of which is provided with a lens. Because the integrated binocular camera includes two left and right lenses, the acquired binocular video stream is also two video streams, namely, a video stream collected by the left lens and a video stream collected by the right lens.

[0054] Step S220: obtaining real-time depth information of the scene corresponding to the binocular video stream according to the binocular video stream.

[0055] In this step, the real-time depth information of the scene corresponding to the above-mentioned binocular video stream may be the real-time depth information of the scene at each moment corresponding to each pair of image frames of the binocular video stream. Among them, the above-mentioned image frame pair is a pair of image frames synchronously captured by the left and right lenses in the integrated binocular camera. The real-time depth information of the scene may be the vertical distance from each pixel point in the scene to the imaging plane of the camera. Specifically, it may be the vertical distance from each pixel point in the scene where the target object in the binocular video stream is located to the imaging plane of the integrated binocular camera. This step obtains the real-time depth information of the scene corresponding to the binocular video stream, so as to accurately determine the motion trajectory of the target object in the three-dimensional scene according to the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream.

[0056] Step S230, generating a motion trajectory of the target object in the binocular video stream based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream.

[0057] In this step, the real-time information of the target object in the binocular video stream in the three-dimensional scene is determined through the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, and then based on the real-time information of the target object in the binocular video stream in the three-dimensional scene, the motion trajectory of the target object in the three-dimensional scene is accurately judged to obtain the motion trajectory of the target object in the binocular video stream.

[0058] In the above steps S210 to S230, the real-time depth information of the scene corresponding to the binocular video stream is obtained through the binocular video stream, and then the motion trajectory of the target object in the three-dimensional scene is accurately determined according to the real-time binocular video stream and the real-time depth information of the scene, and the motion trajectory of the target object in the binocular video stream is obtained. By obtaining the accurate motion trajectory, the parabola point can be accurately located according to the clear motion trajectory, which solves the problem that the parabola point cannot be accurately located due to the inaccurate trajectory path of the parabola.

[0059] In one embodiment, the above step S230, based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, generates a motion trajectory of the target object in the binocular video stream, including the following steps:

[0060] Step S232, based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, determine the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene.

[0061] The above-mentioned real-time motion information may include real-time motion direction and real-time motion speed. The above-mentioned determination of the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream may be based on the binocular video stream, determining the position information of the target object in the binocular video stream in each image frame, and then determining the position, motion direction and motion speed of the target object at each moment according to the position information of the target object in each image frame, combining the position, motion direction and motion speed of the target object at each moment with the real-time depth information of the scene, and obtaining the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene.

[0062] Step S234, generating a motion trajectory of the target object in the binocular video stream based on the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene.

[0063] In this step, the above-mentioned generation of the motion trajectory of the target object in the binocular video stream based on the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene can be performed by first estimating the motion trajectory of the target object in the subsequent binocular video stream based on the real-time position information and real-time motion information of the target object in the current binocular video stream in the three-dimensional scene, and then, according to the estimated motion trajectory of the target object in the subsequent binocular video stream, adjusting the area of ​​interest of the image frame of the video stream to be captured, so that the encoding quality of the detail area of ​​the motion trajectory of the target object in the binocular video stream captured subsequently is higher, so that the movement trajectory of the target object in the binocular video stream is more accurate.

[0064] In the above steps S232 to S234, the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene are determined through the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, and then the motion trajectory of the target object in the subsequent binocular video stream is accurately estimated based on the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene. Then, based on the estimated motion trajectory of the target object in the subsequent binocular video stream, the region of interest of the image frame of the binocular video stream is adjusted to generate a more accurate motion trajectory of the target object in the binocular video stream.

[0065] Further, in one embodiment, the above step S234, based on the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene, generates a motion trajectory of the target object in the binocular video stream, including the following steps:

[0066] Step S2342, based on the real-time position information and real-time motion information of the target object in the three-dimensional scene in the binocular video stream, determine the motion trajectory estimation information of the target object.

[0067] The motion trajectory prediction information of the target object mentioned above may be the real-time position prediction information of the motion trajectory of the target object in the subsequent binocular video stream. Therefore, the motion trajectory prediction information of the target object mentioned above includes the estimated position of the target object in the three-dimensional scene, and the shooting time corresponding to the estimated position. The above-mentioned determination of the motion trajectory prediction information of the target object based on the real-time position information and real-time motion information of the target object in the three-dimensional scene in the binocular video stream may be based on the real-time position information and real-time motion information of the target object in the three-dimensional scene in the binocular video stream that has been captured, and the motion trajectory of the target object is estimated in the three-dimensional scene using a parabolic motion equation or a curve fitting method, thereby obtaining the real-time position estimation information of the motion trajectory of the target object in the subsequent binocular video stream.

[0068] Step S2344: dynamically adjust the region of interest corresponding to the estimated motion trajectory information of the target object in the binocular video stream based on the estimated motion trajectory information of the target object.

[0069] Among them, the region of interest may be the region of interest of each image frame in the binocular video stream. The above-mentioned dynamically adjusting the region of interest corresponding to the estimated motion trajectory information of the target object in the binocular video stream based on the estimated motion trajectory information of the target object may be, in the process of acquiring the binocular video stream using the integrated binocular camera, determining the estimated position of the target object based on the estimated motion trajectory information of the target object, and dynamically adjusting the encoding parameters of the integrated binocular camera, so that the region of interest of the image frame captured by the integrated binocular camera matches the real-time position of the target object, and the area range of the region of interest includes the area range of the target object, thereby realizing the dynamic adjustment of the region of interest corresponding to the estimated motion trajectory information of the target object in the binocular video stream.

[0070] In this step, in the process of obtaining the binocular video stream using the integrated binocular camera, the encoding parameters of the integrated binocular camera are dynamically adjusted so that the region of interest of the image frame captured by the integrated binocular camera matches the real-time position of the target object, so that the encoding quality of the detail region of the target object in each image frame in the binocular video stream is higher, and the motion trajectory of the target object in the binocular video stream is generated with higher accuracy. The picture-in-picture encoding of the parabolic trajectory details of the target object is achieved, and the detection of the detail trajectory in the picture-in-picture encoding is more accurate, which can solve the problem that the encoding quality of the video stream and the captured image of the target object's trajectory path detail area is not clear enough.

[0071] Step S2346: Generate a motion trajectory of the target object in the binocular video stream based on the adjusted region of interest.

[0072] In this step, the adjusted region of interest is used to match the region of interest of the image frame captured by the integrated binocular camera with the real-time position of the target object, thereby obtaining a binocular video stream with higher encoding quality for the detailed region of the motion trajectory of the target object.

[0073] The above steps S2342 to S2346 determine the estimated motion trajectory information of the target object through the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene, and then dynamically adjust the region of interest corresponding to the estimated motion trajectory information of the target object in the binocular video stream based on the estimated motion trajectory information of the target object, so that the region of interest of the image frame captured by the integrated binocular camera matches the real-time position of the target object, and obtains the motion trajectory of the target object in the binocular video stream with higher encoding quality of the detail area. The accuracy of the motion trajectory of the target object in the binocular video stream is higher, the motion trajectory is clearer, and it is convenient to accurately locate the parabola point according to the clear motion trajectory, solving the problem that the parabola point cannot be accurately located due to the inaccurate trajectory path of the parabola.

[0074] In addition, in one embodiment, the above step S232, based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, determines the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene, including the following steps:

[0075] Step S2322: Based on the binocular video stream, determine the position of the target object in each image frame in the binocular video stream.

[0076] The above-mentioned determination of the position of the target object in each image frame in the binocular video stream may be performed by using a preset target object recognition method to determine the position of the target object in each image frame in the binocular video stream. The above-mentioned preset target object recognition method may be one or more of directly identifying the target object through a preset target recognition model, detecting the target object using an optical flow method, detecting the target object using an inter-frame difference method, detecting the target object using a background subtraction method, etc. The above-mentioned preset target object recognition method may be one or more of them, or may be other target object recognition methods. This embodiment is not specifically limited here, as long as the position of the target object in each image frame in the binocular video stream can be determined. At this time, the position of the target object in each image frame may be represented in the form of a two-dimensional coordinate on a plane on the image frame.

[0077] Step S2324, based on the position of the target object in each image frame in the binocular video stream, determine the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream.

[0078] In this step, the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream are determined based on the position of the target object in each image frame in the binocular video stream. The change in position information of the target object in the previous and next frames in the image frames can be determined according to the position information of the target object in the binocular video stream in each image frame. Then, based on the change in position information of the target object in the previous and next frames in the image frames and the frame rate of image frame acquisition, the position, movement direction and movement speed of the target object at each moment are determined, that is, the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream are obtained.

[0079] Step S2326, based on the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream, and the real-time depth information of the scene corresponding to the binocular video stream, determine the real-time position information and real-time movement information of the target object in the binocular video stream in the three-dimensional scene.

[0080] Among them, the real-time position information of the target object in the three-dimensional scene can be represented by a position representation method of the three-dimensional scene, specifically, the real-time position of the target object in the three-dimensional scene can be represented by using a Cartesian coordinate system, a spherical coordinate system, Euler angles, a grid representation, a cylindrical coordinate system, a point cloud representation, a voxel representation or a homogeneous coordinate system.

[0081] In the above steps S2322 to S2326, based on the binocular video stream, the position information of the target object in each image frame in the binocular video stream is determined, and then the position, movement direction and movement speed of the target object at each moment are determined according to the position information of the target object in each image frame, and the position, movement direction and movement speed of the target object at each moment are combined with the real-time depth information of the scene to obtain the real-time position information and real-time movement information of the target object in the binocular video stream in the three-dimensional scene. It is convenient to accurately estimate the movement trajectory of the target object in the subsequent binocular video stream according to the real-time position information and real-time movement information of the target object in the binocular video stream in the three-dimensional scene.

[0082] Further, in one embodiment, the above step S2324, based on the position of the target object in each image frame in the binocular video stream, determines the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream, may include the following steps:

[0083] Step S23242, based on the position of the target object in each image frame in the binocular video stream, determine the positional relationship between the position of the target object in the current image frame in the binocular video stream and the positions of the target objects in the previous and next image frames.

[0084] Step S23244, based on the positional relationship between the position of the target object in the current image frame in the binocular video stream and the positions of the target objects in the previous and next image frames, determine the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream.

[0085] Wherein, the above-mentioned determination of the real-time position, real-time motion direction and real-time motion speed of the target object in the binocular video stream based on the positional relationship between the position of the target object in the current image frame in the binocular video stream and the position of the target object in the previous and next image frames can be based on the positional relationship between the position of the target object in the current image frame in the binocular video stream and the position of the target object in the previous and next image frames, and the frame rate of the image frame acquisition, to determine the real-time position, real-time motion direction and real-time motion speed of the target object in the binocular video stream. Specifically, the position change amount and the direction change amount of the target object between the previous and next frames can be determined based on the positional relationship between the position of the target object in the current image frame in the binocular video stream and the position of the target object in the previous and next image frames, and the time corresponding to the position change amount and the direction change amount of the target object can be obtained based on the frame rate of the image frame acquisition, and further, the real-time position, real-time motion direction and real-time motion speed of the target object in the binocular video stream can be obtained based on the position change amount of the target object, the direction change amount of the target object and the time corresponding to the position change amount and the direction change amount.

[0086] In the above steps S23242 to S23244, the positional relationship between the position of the target object in the current image frame in the binocular video stream and the position of the target object in the previous and next image frames is determined through the position of the target object in each image frame in the binocular video stream, and then the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream are obtained according to the positional relationship between the position of the target object in the current image frame in the binocular video stream and the position of the target object in the previous and next image frames. The real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream are obtained to facilitate the combination with the real-time depth information of the scene corresponding to the binocular video stream, and determine the real-time position information and real-time movement information of the target object in the binocular video stream in the three-dimensional scene.

[0087] In addition, in one embodiment, the above step S220, obtaining the scene real-time depth information corresponding to the binocular video stream according to the binocular video stream, includes:

[0088] Step S222: determining, according to the binocular video stream, a real-time binocular parallax relationship between left and right lenses of an integrated binocular camera corresponding to the binocular video stream.

[0089] The above-mentioned determination of the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream according to the binocular video stream may be based on the binocular video stream, determining each image frame pair of the binocular video stream, and then determining the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream based on each image frame pair. Wherein, the above-mentioned image frame pair is a pairing of image frames synchronously captured by the left and right lenses of the integrated binocular camera. The above-mentioned determination of the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream based on each image frame pair may be based on each image frame pair, using a preset binocular parallax calculation method, to obtain the binocular parallax between the left and right lenses of the integrated binocular camera corresponding to each image frame pair, and then determining the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream. The above-mentioned preset binocular disparity calculation method can be a stereo matching algorithm, specifically local matching, global matching, semi-global matching, or one or more methods in a stereo matching algorithm based on deep learning. As long as the real-time binocular disparity relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream can be obtained through the preset binocular disparity calculation method, the specific binocular disparity calculation method is not specifically limited in this embodiment.

[0090] Step S224, based on the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream, obtain the real-time depth information of the scene corresponding to the binocular video stream.

[0091] The above-mentioned real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream is used to obtain the real-time depth information of the scene corresponding to the binocular video stream. The real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream can be obtained by using a preset depth information calculation method. The above-mentioned preset depth information calculation method can be one or more of a geometric method, a camera parameter derivation method, etc. The specific depth information calculation method is not specifically limited in this embodiment, as long as the real-time depth information of the scene corresponding to the binocular video stream can be obtained through the real-time binocular parallax relationship between the left and right lenses.

[0092] In the above steps S222 to S224, the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream is determined through the binocular video stream, and then the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream is obtained. The real-time depth information of the scene corresponding to the binocular video stream is obtained, so as to facilitate the subsequent accurate judgment of the motion trajectory of the target object in the three-dimensional scene according to the real-time depth information of the scene corresponding to the binocular video stream.

[0093] The present embodiment is described and illustrated below through preferred embodiments.

[0094] Figure 3 : is a flow chart of a motion trajectory generation method provided by a preferred embodiment of the present application. Figure 3 As shown, the motion trajectory generation method includes the following steps:

[0095] Step 310, obtaining a binocular video stream; the binocular video stream is a video stream synchronously collected by the left and right lenses of the integrated binocular camera;

[0096] Step 320, determining a real-time binocular parallax relationship between left and right lenses of an integrated binocular camera corresponding to the binocular video stream according to the binocular video stream;

[0097] Step 330, obtaining real-time depth information of the scene corresponding to the binocular video stream based on the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream;

[0098] Step 340, determining the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream;

[0099] Step 350, determining motion trajectory estimation information of the target object based on the real-time position information and real-time motion information of the target object in the three-dimensional scene in the binocular video stream;

[0100] Step 360, dynamically adjusting the region of interest corresponding to the estimated motion trajectory information of the target object in the binocular video stream based on the estimated motion trajectory information of the target object;

[0101] Step 370: Generate a motion trajectory of the target object in the binocular video stream based on the adjusted region of interest.

[0102] In the above steps S310 to S370, the real-time depth information of the scene corresponding to the binocular video stream is obtained through the binocular video stream, and then the motion trajectory of the target object in the three-dimensional scene is accurately determined according to the real-time binocular video stream and the real-time depth information of the scene, and the motion trajectory of the target object in the binocular video stream is obtained. By obtaining the accurate motion trajectory, the parabola point can be accurately located according to the clear motion trajectory, which solves the problem that the parabola point cannot be accurately located due to the inaccurate trajectory path of the parabola.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, a motion trajectory generating device is also provided in this embodiment, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0105] In one embodiment, Figure 4 is a structural block diagram of a motion trajectory generating device provided by an embodiment of the present application, such as Figure 4 As shown, the motion trajectory generating device comprises:

[0106] An acquisition module 42 is used to synchronously acquire a video stream using left and right lenses in an integrated binocular camera;

[0107] The calculation module 44 is used to determine the position of the target object in each image frame in the binocular video stream based on the binocular video stream; determine the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream; calculate the real-time position information and real-time movement information of the target object in the binocular video stream in the three-dimensional scene based on the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream;

[0108] And an encoding module 46 is used to determine the motion trajectory prediction information of the target object based on the real-time position information and real-time motion information of the target object in the three-dimensional scene in the binocular video stream; based on the motion trajectory prediction information of the target object, dynamically adjust the area of ​​interest in the binocular video stream corresponding to the motion trajectory prediction information of the target object.

[0109] The motion trajectory generating device uses the left and right lenses in the integrated binocular camera to synchronously collect video streams to obtain binocular video streams, and then determines the position of the target object in each image frame in the binocular video stream according to the binocular video stream, and determines the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream; calculates the real-time position information and real-time movement information of the target object in the binocular video stream in a three-dimensional scene based on the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream, and the real-time depth information of the scene corresponding to the binocular video stream, determines the motion trajectory prediction information of the target object based on the real-time position information and real-time movement information of the target object in the binocular video stream in the three-dimensional scene, and dynamically adjusts the interest area corresponding to the motion trajectory prediction information of the target object in the binocular video stream based on the motion trajectory prediction information of the target object, so that the interest area of ​​the image frame captured by the integrated binocular camera matches the real-time position of the target object, and obtains the motion trajectory of the target object in the binocular video stream with higher coding quality of the detail area. The motion trajectory of the target object in the binocular video stream is more accurate and clearer, making it easier to accurately locate the parabola point based on the clear motion trajectory, thus solving the problem of being unable to accurately locate the parabola point due to the inaccurate trajectory path of the parabola.

[0110] Based on the same inventive concept, a motion trajectory generation system is also provided in this embodiment, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0111] In one embodiment, Figure 5 is a structural block diagram of a motion trajectory generation system provided by an embodiment of the present application, such as Figure 5 As shown, the motion trajectory generation system includes:

[0112] The acquisition module 52 is used to acquire a binocular video stream; the binocular video stream is a video stream synchronously collected by the left and right lenses of the integrated binocular camera;

[0113] A depth information determination module 54 is used to obtain real-time depth information of the scene corresponding to the binocular video stream according to the binocular video stream;

[0114] And a trajectory generation module 56 is used to generate a motion trajectory of a target object in the binocular video stream based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream.

[0115] The above-mentioned motion trajectory generation system obtains the real-time depth information of the scene corresponding to the binocular video stream through the binocular video stream, and then accurately judges the motion trajectory of the target object in the three-dimensional scene based on the real-time binocular video stream and the real-time depth information of the scene, and obtains the motion trajectory of the target object in the binocular video stream. By obtaining accurate motion trajectories, it is possible to accurately locate the parabola point based on clear motion trajectories, solving the problem of being unable to accurately locate the parabola point due to the inaccurate trajectory path of the parabola.

[0116] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0117] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the motion trajectory generation methods in the above embodiments is implemented.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A motion trajectory generation method, characterized in that: The method comprises: Obtain a binocular video stream; the binocular video stream is a video stream synchronously collected by left and right lenses of an integrated binocular camera; According to the binocular video stream, real-time depth information of the scene corresponding to the binocular video stream is obtained; Based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, a motion trajectory of the target object in the binocular video stream is generated.

2. The motion trajectory generation method according to claim 1, characterized in that: The step of generating a motion trajectory of a target object in the binocular video stream based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream comprises: Based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream, determining the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene; Based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene, a motion trajectory of the target object in the binocular video stream is generated.

3. The motion trajectory generation method according to claim 2, characterized in that: The step of generating a motion track of the target object in the binocular video stream based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene comprises: Determine motion trajectory estimation information of the target object based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene; Based on the motion trajectory estimation information of the target object, dynamically adjusting the region of interest in the binocular video stream corresponding to the motion trajectory estimation information of the target object; Based on the adjusted region of interest, a motion trajectory of the target object in the binocular video stream is generated.

4. The motion trajectory generation method according to claim 2, characterized in that: The method of determining the real-time position information and real-time motion information of the target object in the binocular video stream in the three-dimensional scene based on the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream includes: Based on the binocular video stream, determining the position of the target object in each image frame in the binocular video stream; Determine the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream; Based on the real-time position, the real-time movement direction and the real-time movement speed of the target object in the binocular video stream, and the real-time depth information of the scene corresponding to the binocular video stream, the real-time position information and the real-time movement information of the target object in the binocular video stream in the three-dimensional scene are determined.

5. The motion trajectory generation method according to claim 4, characterized in that: The method of determining the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream comprises: Based on the position of the target object in each image frame in the binocular video stream, determining a positional relationship between the position of the target object in the current image frame in the binocular video stream and the positions of the target objects in the previous and next image frames; Based on the positional relationship between the position of the target object in the current image frame in the binocular video stream and the position of the target object in the previous and next image frames, the real-time position, the real-time movement direction and the real-time movement speed of the target object in the binocular video stream are determined.

6. The motion trajectory generation method according to claim 1, characterized in that: The step of obtaining the scene real-time depth information corresponding to the binocular video stream according to the binocular video stream includes: Determine, according to the binocular video stream, a real-time binocular parallax relationship between left and right lenses of the integrated binocular camera corresponding to the binocular video stream; Based on the real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream, the real-time depth information of the scene corresponding to the binocular video stream is obtained.

7. The motion trajectory generation method according to claim 6, characterized in that: The step of determining, according to the binocular video stream, a real-time binocular parallax relationship between left and right lenses of the integrated binocular camera corresponding to the binocular video stream comprises: Based on the binocular video stream, determining each pair of image frames of the binocular video stream; the image frame pair is a pair of image frames synchronously captured by left and right lenses in the integrated binocular camera; Based on each of the image frame pairs, a real-time binocular parallax relationship between the left and right lenses of the integrated binocular camera corresponding to the binocular video stream is determined.

8. A motion trajectory generating device, characterized in that: The device comprises: An acquisition module, used for synchronously acquiring video streams using left and right lenses of an integrated binocular camera; A calculation module is used to determine the position of a target object in each image frame in the binocular video stream based on the binocular video stream; determine the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream based on the position of the target object in each image frame in the binocular video stream; calculate the real-time position information and real-time movement information of the target object in the binocular video stream in a three-dimensional scene based on the real-time position, real-time movement direction and real-time movement speed of the target object in the binocular video stream and the real-time depth information of the scene corresponding to the binocular video stream; And an encoding module, which is used to determine the motion trajectory prediction information of the target object based on the real-time position information and the real-time motion information of the target object in the binocular video stream in the three-dimensional scene; based on the motion trajectory prediction information of the target object, dynamically adjust the area of ​​interest in the binocular video stream corresponding to the motion trajectory prediction information of the target object.

9. A motion trajectory generation system, characterized in that: The system comprises: An acquisition module is used to acquire a binocular video stream; the binocular video stream is a video stream synchronously collected by the left and right lenses of the integrated binocular camera; A depth information determination module, used to obtain real-time depth information of the scene corresponding to the binocular video stream according to the binocular video stream; And a trajectory generation module is used to generate a motion trajectory of a target object in the binocular video stream based on the binocular video stream and real-time depth information of the scene corresponding to the binocular video stream.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the motion trajectory generation method according to any one of claims 1 to 7 are implemented.

Images