A moving target detection and tracking method, device, equipment, medium and product

Abstract

The application discloses a kind of active target detection and tracking method, device, equipment, medium and product, it is related to video image analysis field.The method comprises: acquiring 360 degree panoramic video image and the motion attitude data of panoramic camera collected by panoramic camera;According to motion attitude data, three degrees of freedom coordinate normalization is carried out to each frame panoramic image in 360 degree panoramic video image;The panoramic residual image of each adjacent two frames panoramic image in normalized panoramic video image is calculated, and the part corresponding to panoramic residual image in current frame panoramic image is determined as target area;Spherical convolution is carried out to target area, and the active target corresponding to current frame panoramic image is obtained;The active target corresponding to current frame panoramic image is tracked, and the active target tracking result of 360 degree panoramic video image is obtained.The application can realize the detection and tracking of active target in the scene of three degrees of freedom rotation motion of yaw, pitch, roll of 360 degree panoramic camera.

Description

Technical Field

[0001] This application relates to the field of video image analysis, and in particular to a method, apparatus, device, medium, and product for detecting and tracking moving targets. Background Technology

[0002] Target detection and tracking are core functions of intelligent video image analysis. Target detection based solely on a single frame can only identify objects present within the image, but cannot determine whether the target is active (moving) or stationary. Continuous detection across multiple frames is necessary to determine the target's motion attributes based on changes in its position across frames, and to simultaneously perform continuous target tracking. Similarly, for panoramic image scenes, determining whether a target is active also requires continuous target detection and cross-frame tracking processing based on multiple panoramic video frames.

[0003] Currently, relevant target detection methods typically only perform target detection on single-frame panoramic images, neglecting target detection and continuous tracking across multiple frames. Furthermore, when a 360-degree panoramic camera undergoes yaw, pitch, and roll (3DoF) rotational motion, the acquired panoramic image experiences an overall attitude shift. Distinguishing and detecting real moving targets and achieving continuous tracking across frames under such global motion conditions presents significant challenges. Therefore, how to detect and track moving targets in scenarios involving 3DoF rotational motion of a 360-degree panoramic camera has become a pressing issue. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, medium, and product for detecting and tracking moving targets, which can realize the detection and tracking of moving targets in scenarios where a 360-degree panoramic camera undergoes three degrees of freedom rotational motion (yaw, pitch, and roll).

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for detecting and tracking moving targets, including: Acquire 360-degree panoramic video images captured by a panoramic camera; Acquire motion attitude data from the panoramic camera; Based on the motion posture data, the three-degree-of-freedom coordinates of each frame of the 360-degree panoramic video image are normalized to obtain the normalized panoramic video image. Calculate the panoramic residual image of each two adjacent panoramic images in the normalized panoramic video image. For the current panoramic image, determine the part of the current panoramic image that corresponds to the panoramic residual image as the target region. For the target region in the current frame panoramic image, perform spherical convolution on the target region to obtain the active target corresponding to the current frame panoramic image; The moving targets corresponding to the current frame panoramic image are tracked to obtain the moving target tracking results of the 360-degree panoramic video image.

[0006] In one embodiment, acquiring the motion pose data of the panoramic camera specifically includes: A 3DoF sensor is used to sense the current motion posture of the panoramic camera and obtain motion posture data.

[0007] In one embodiment, based on the motion posture data, each frame of the 360-degree panoramic video image is normalized using three degrees of freedom coordinates to obtain a normalized panoramic video image, specifically including: For any frame of the 360-degree panoramic video image, determine the pixel coordinates of the frame and determine the three-degree-of-freedom three-axis rotation angle of the frame based on the motion posture data. For any frame of the 360-degree panoramic video image, the pixel coordinates of the frame are used as the original position data. The pixels of the frame are mapped to the camera's normalized imaging plane, and the mapped pixels are converted into the three-dimensional spherical coordinates of the panoramic camera in the current motion posture. For any frame of the 360-degree panoramic video image, construct the inverse rotation matrix of the Bursa model based on the three degrees of freedom and three-axis rotation angle of the panoramic image of that frame, and restore the three-dimensional spherical coordinates to the reference coordinate system based on the inverse rotation matrix of the Bursa model to obtain the restored three-dimensional coordinates; For any frame of the 360-degree panoramic video image, the three-dimensional coordinates of the restored panoramic image are back-projected and scaled to obtain the normalized two-dimensional pixel coordinates of the panoramic image; the normalized two-dimensional pixel coordinates of all frames of panoramic images constitute the normalized panoramic video image.

[0008] In one embodiment, calculating the panoramic residual image of every two adjacent panoramic frames in the normalized panoramic video image specifically includes: The panoramic residual image of the next frame is obtained by subtracting each two adjacent panoramic frames in the normalized panoramic video image.

[0009] In one embodiment, the 3DoF sensor includes an inertial measurement unit.

[0010] In one embodiment, the inertial measurement unit includes at least a three-axis gyroscope.

[0011] Secondly, this application provides a moving target detection and tracking device, comprising: The image acquisition unit is used to acquire 360-degree panoramic video images captured by the panoramic camera; The data acquisition unit is used to acquire motion attitude data from the panoramic camera. The coordinate normalization unit is used to perform three-degree-of-freedom coordinate normalization on each frame of the 360-degree panoramic video image based on the motion posture data, so as to obtain the normalized panoramic video image. The target region determination unit is used to calculate the panoramic residual image of each two adjacent panoramic images in the normalized panoramic video image. For the current frame panoramic image, the part of the current frame panoramic image corresponding to the panoramic residual image is determined as the target region. The active target detection unit is used to perform spherical convolution on the target region in the current frame panoramic image to obtain the active target corresponding to the current frame panoramic image; The moving target tracking unit is used to track the moving targets corresponding to the current frame panoramic image and obtain the moving target tracking results of the 360-degree panoramic video image.

[0012] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the moving target detection and tracking method described in any one of the above.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the moving target detection and tracking method described in any one of the above descriptions.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the active target detection and tracking method described in any one of the above descriptions.

[0015] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, apparatus, device, medium, and product for detecting and tracking moving targets. Based on motion attitude data, each frame of the panoramic image in a 360-degree panoramic video image is normalized to three degrees of freedom coordinates, ensuring that the normalized panoramic video image remains within a unified coordinate system under any three-degree-of-freedom rotational transformation of the panoramic camera. By calculating the panoramic residual image of each adjacent two frames of the normalized panoramic video image, the detection and tracking of moving targets in the 360-degree panoramic video image are achieved. This application realizes the detection and tracking of moving targets in scenarios where the 360-degree panoramic camera undergoes yaw, pitch, and roll three-degree-of-freedom rotational motion. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an active target detection and tracking method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the functional modules of an active target detection and tracking device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] When a panoramic camera performs three-degree-of-freedom (yaw, pitch, and roll) motion, the entire image captured by the camera is in motion. Detecting and tracking moving targets within this moving image presents a challenge. The purpose of this application is to provide a method, apparatus, device, medium, and product for detecting and tracking moving targets, enabling the detection and tracking of moving targets when a 360-degree panoramic camera performs three-degree-of-freedom (yaw, pitch, and roll) motion.

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] In one exemplary embodiment, such as Figure 1 As shown, a method for detecting and tracking moving targets is provided, including: Step 101: Acquire 360-degree panoramic video images captured by the panoramic camera.

[0022] Step 102: Obtain motion attitude data from the panoramic camera.

[0023] Step 103: Based on the motion posture data, normalize the three-degree-of-freedom coordinates of each frame of the 360-degree panoramic video image to obtain the normalized panoramic video image.

[0024] Step 104: Calculate the panoramic residual image of each two adjacent panoramic images in the normalized panoramic video image. For the current panoramic image, determine the part in the current panoramic image that corresponds to the panoramic residual image as the target region.

[0025] Step 105: For the target region in the current frame panoramic image, perform spherical convolution on the target region to obtain the active target corresponding to the current frame panoramic image.

[0026] Step 106: Track the moving target corresponding to the current frame panoramic image to obtain the moving target tracking result of the 360-degree panoramic video image.

[0027] In another exemplary embodiment of this application, step 102 specifically includes: using a 3DoF sensor to sense the current motion posture of the panoramic camera and obtaining motion posture data.

[0028] The 3DoF sensor includes an inertial measurement unit (IMU). In one embodiment, the inertial measurement unit includes at least a three-axis gyroscope. The three-axis gyroscope acquires motion attitude data from the panoramic camera; the motion attitude data includes pitch, roll, yaw, and rotation attitudes.

[0029] As an optional implementation, the inertial measurement unit further includes a triaxial accelerometer and a triaxial magnetometer. The triaxial accelerometer is used to correct gyro drift by relying on the gravity vector, either statically or at low speeds, to accurately calculate pitch and roll angles; the triaxial magnetometer is used to sense the Earth's magnetic field and provide absolute orientation to correct yaw drift.

[0030] In another exemplary embodiment of this application, step 103 specifically includes: For any frame of the 360-degree panoramic video image, determine the pixel coordinates of the frame and determine the three-degree-of-freedom three-axis rotation angle of the frame based on the motion posture data.

[0031] For any frame of the 360-degree panoramic video image, the pixel coordinates of the frame are used as the original position data. The pixels of the frame are mapped to the camera's normalized imaging plane, and the mapped pixels are converted into three-dimensional spherical coordinates of the panoramic camera in its current motion posture.

[0032] For any frame of the 360-degree panoramic video image, construct the inverse rotation matrix of the Bursa model based on the three degrees of freedom and three-axis rotation angle of the panoramic image of that frame, and restore the three-dimensional spherical coordinates to the reference coordinate system based on the inverse rotation matrix of the Bursa model to obtain the restored three-dimensional coordinates.

[0033] For any frame of the 360-degree panoramic video image, the three-dimensional coordinates of the restored panoramic image are back-projected and scaled to obtain the normalized two-dimensional pixel coordinates of the panoramic image; the normalized two-dimensional pixel coordinates of all frames of panoramic images constitute the normalized panoramic video image.

[0034] In another exemplary embodiment of this application, step 104 specifically includes: subtracting every two adjacent panoramic frames in the normalized panoramic video image, with the previous frame as the reference frame, and the image obtained after subtraction as the panoramic residual image of the next panoramic frame.

[0035] In another exemplary embodiment of this application, step 105 specifically includes: for a target region in any frame of panoramic image, performing spherical convolution on the target region to obtain a spherical convolution thermal feature map, and determining the active target corresponding to the frame of panoramic image based on the spherical convolution thermal feature map.

[0036] Specifically, based on the spherical convolutional thermal feature map, motion candidate spherical blocks are obtained by segmentation using the convolutional response threshold. Then, by combining spherical geometric size constraints and multi-frame temporal convolutional feature continuity verification, pseudo-motion regions caused by illumination and noise are filtered out. Finally, target classification and pixel plane mapping are completed on the spherical connected feature blocks that retain stable high convolutional responses, so as to achieve accurate detection of moving targets in panoramic images.

[0037] In another exemplary embodiment of this application, step 106 specifically includes: for moving targets detected in two adjacent panoramic images, extracting the two-dimensional texture features and bounding box geometric features of each moving target in the two frames to obtain target features; performing feature matching based on feature similarity and coordinate space distance, and combining the Hungarian algorithm to complete the pairing of the same moving target between the two frames and assigning a unique tracking identifier; then combining the target position change between the two frames to predict the estimated position of the target in subsequent frames through Kalman filtering to deal with missed detections; subsequently, taking the current frame target set that has completed matching and has a tracking identifier as the new previous frame target set, and repeating the above two-frame matching association process with the moving targets detected in the next frame, thereby realizing continuous target tracking under multi-frame panoramic images and obtaining the moving target tracking result of 360-degree panoramic video images.

[0038] The implementation process of the moving target detection and tracking method in this embodiment is as follows: First, read 360-degree panoramic video images; Second, read the values ​​from the 3DoF sensor to perceive the current motion posture, and use the 3DoF sensor values ​​to normalize the coordinates of each frame in the panoramic video image using the 3DoF coordinate normalization unit. This ensures that regardless of how the panoramic camera rotates in the 3DoF dimension, the final panoramic video image is within a unified coordinate system. Furthermore, this projection method has low computational cost and low power consumption, making it suitable for battery-powered operation; Third, perform moving target detection on the 3DoF-normalized panoramic video image. Specifically, subtract two consecutive frames to obtain a panoramic residual image, which significantly reduces the computational cost of target detection. Target detection is only performed on the portion of the panoramic image corresponding to the residual region. Spherical convolution is then performed on the existing residual region in the current panoramic image to detect moving targets (such as a flying drone); Fourth, track moving targets based on the detected moving targets across multiple frames of panoramic images.

[0039] This embodiment realizes the detection and tracking of moving targets in scenarios where the 3DoF motion of the panoramic camera is yaw, pitch, and roll.

[0040] Based on the same inventive concept, this application also provides an active target detection and tracking device for implementing the active target detection and tracking method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more active target detection and tracking device embodiments provided below can be found in the limitations of the active target detection and tracking method described above, and will not be repeated here.

[0041] In one exemplary embodiment, such as Figure 2As shown, a moving target detection and tracking device is provided, comprising: an image acquisition unit 201 for acquiring 360-degree panoramic video images captured by a panoramic camera; a data acquisition unit 202 for acquiring motion attitude data of the panoramic camera; a coordinate normalization unit 203 for performing three-degree-of-freedom coordinate normalization on each frame of the 360-degree panoramic video image based on the motion attitude data to obtain a normalized panoramic video image; a target region determination unit 204 for calculating the panoramic residual image of every two adjacent frames of the normalized panoramic video image, and for the current frame of the panoramic image, determining the portion of the current frame of the panoramic image corresponding to the panoramic residual image as the target region; a moving target detection unit 205 for performing spherical convolution on the target region in the current frame of the panoramic image to obtain the moving target corresponding to the current frame of the panoramic image; and a moving target tracking unit 206 for tracking the moving target corresponding to the current frame of the panoramic image to obtain the moving target tracking result of the 360-degree panoramic video image.

[0042] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores the results of moving target detection and tracking of 360-degree panoramic video images. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a moving target detection and tracking method.

[0043] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include, for example, [the following is a list of possible additional structures]. Figure 3The embodiments show more or fewer components, combinations of certain components, or different component arrangements. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above-described method embodiments.

[0044] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0045] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0046] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0047] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0048] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0049] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0050] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of moving object detection and tracking, characterized by, include: Acquire 360-degree panoramic video images captured by a panoramic camera; Acquire motion attitude data from the panoramic camera; Based on the motion posture data, the three-degree-of-freedom coordinates of each frame of the 360-degree panoramic video image are normalized to obtain the normalized panoramic video image. Calculate the panoramic residual image of each two adjacent panoramic images in the normalized panoramic video image. For the current panoramic image, determine the part of the current panoramic image that corresponds to the panoramic residual image as the target region. For the target region in the current frame panoramic image, perform spherical convolution on the target region to obtain the active target corresponding to the current frame panoramic image; The moving targets corresponding to the current frame panoramic image are tracked to obtain the moving target tracking results of the 360-degree panoramic video image.

2. The method for detecting and tracking moving targets according to claim 1, characterized in that, Acquire motion pose data from the panoramic camera, specifically including: A 3DoF sensor is used to sense the current motion posture of the panoramic camera and obtain motion posture data.

3. The method for detecting and tracking moving targets according to claim 1, characterized in that, Based on the motion posture data, each frame of the 360-degree panoramic video image is normalized using three degrees of freedom coordinates to obtain a normalized panoramic video image, specifically including: For any frame of the 360-degree panoramic video image, determine the pixel coordinates of the frame and determine the three-degree-of-freedom three-axis rotation angle of the frame based on the motion posture data. For any frame of the 360-degree panoramic video image, the pixel coordinates of the frame are used as the original position data. The pixels of the frame are mapped to the camera's normalized imaging plane, and the mapped pixels are converted into the three-dimensional spherical coordinates of the panoramic camera in the current motion posture. For any frame of the 360-degree panoramic video image, construct the inverse rotation matrix of the Bursa model based on the three degrees of freedom and three-axis rotation angle of the panoramic image of that frame, and restore the three-dimensional spherical coordinates to the reference coordinate system based on the inverse rotation matrix of the Bursa model to obtain the restored three-dimensional coordinates; For any frame of the 360-degree panoramic video image, the three-dimensional coordinates of the restored panoramic image are back-projected and scaled to obtain the normalized two-dimensional pixel coordinates of the panoramic image; the normalized two-dimensional pixel coordinates of all frames of panoramic images constitute the normalized panoramic video image.

4. The method for detecting and tracking moving targets according to claim 1, characterized in that, Calculate the panoramic residual image between every two adjacent frames of the normalized panoramic video image, specifically including: The panoramic residual image of the next frame is obtained by subtracting each two adjacent panoramic frames in the normalized panoramic video image.

5. The method for detecting and tracking moving targets according to claim 2, characterized in that, The 3DoF sensor includes an inertial measurement unit.

6. The method for detecting and tracking moving targets according to claim 5, characterized in that, The inertial measurement unit includes at least a three-axis gyroscope.

7. A moving target detection and tracking device, characterized in that, include: The image acquisition unit is used to acquire 360-degree panoramic video images captured by the panoramic camera; The data acquisition unit is used to acquire motion attitude data from the panoramic camera. The coordinate normalization unit is used to perform three-degree-of-freedom coordinate normalization on each frame of the 360-degree panoramic video image based on the motion posture data, so as to obtain the normalized panoramic video image. The target region determination unit is used to calculate the panoramic residual image of each two adjacent panoramic images in the normalized panoramic video image. For the current frame panoramic image, the part of the current frame panoramic image corresponding to the panoramic residual image is determined as the target region. The active target detection unit is used to perform spherical convolution on the target region in the current frame panoramic image to obtain the active target corresponding to the current frame panoramic image; The moving target tracking unit is used to track the moving targets corresponding to the current frame panoramic image and obtain the moving target tracking results of the 360-degree panoramic video image.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the moving target detection and tracking method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the active target detection and tracking method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the active target detection and tracking method according to any one of claims 1-6.

Images