Camera posture perception system based on deep learning

Abstract

The present invention provides a camera posture perception system based on deep learning, which uses deep learning methods combined with sensor technology to obtain real-time camera posture data, especially posture data during camera movement. The camera posture perception system includes: an auxiliary circuit, a communication network, a magnetic sensor, a camera, a pan / tilt, a data processing unit, and a communication unit; the camera posture perception system is initialized before use, and the twin neural network is trained to obtain a model through the environmental image captured by the camera; when the system is working, the data processing unit obtains the camera's posture data by inputting the video image captured by the camera into the twin neural network model. The camera posture perception system solves the problem of real-time posture perception of rotatable cameras, has the characteristics of strong real-time performance, accurate data, and easy installation and implementation, and can be widely promoted and applied in public security, factories, transportation, mining and other fields.

Description

Technical Field

[0001] The present invention designs a camera posture perception system based on deep learning, which involves fields such as image acquisition, image feature extraction, deep learning and communication. Background Art

[0002] With the increasing prevalence and application of surveillance equipment in daily life, cameras are playing a vital role in public security, factories, transportation, mining, and other fields. Surveillance cameras come in two types: fixed and rotatable. Fixed cameras have fixed pan and tilt angles, resulting in a limited monitoring range, capturing only image information within a fixed field of view. Rotatable cameras with a pan / tilt (PTZ) head (pan / tilt) can significantly expand the monitoring range. However, because the PTZ head uses open-loop control, the monitoring center can control its rotation but cannot obtain the camera's real-time angle. Even with a specialized PTZ head with closed-loop control, the monitoring center cannot accurately determine the camera's real-time angle if the camera is manually rotated by on-site staff or if other external forces change its angle. Without accurate real-time angle information, the monitoring center cannot accurately determine the monitoring area of ​​the captured image, making it impossible to accurately locate the monitored target using image pattern recognition technology. This severely restricts the development and application of surveillance technology. Therefore, a method and system for real-time posture perception of rotatable cameras are needed. Summary of the Invention

[0003] The present invention designs a camera posture perception system based on deep learning, which uses deep learning methods combined with sensor technology to obtain real-time camera posture data, especially posture data during camera movement. The camera posture perception system includes: an auxiliary circuit, a magnetic sensor, a camera, a pan-tilt head, a data processing unit, and a communication unit; the camera is fixedly mounted on a rotating pan-tilt head for collecting images; the magnetic sensor is used to determine the absolute posture data of the camera when it is stationary; the auxiliary circuit is used to provide power to each part; the data processing unit is used to process the images collected by the camera and the magnetic data collected by the magnetic sensor to obtain real-time absolute camera posture data; the pan-tilt head is used to drive the camera to rotate; the communication unit is responsible for sending video image data and camera posture data, as well as receiving camera setting data and pan-tilt head control data;

[0004] The camera attitude perception system needs to be initialized before use. The initialization process is as follows:

[0005] (1) Capturing image M1 through a camera;

[0006] (2) Magnetic data is collected by a magnetic sensor, and the magnetic data is processed to determine the absolute position data α1 and β1 corresponding to M1; α1 and β1 are the horizontal rotation angle and vertical rotation angle expressed in Euler angles, respectively;

[0007] (3) The PTZ drives the camera to rotate and synchronously collect image M i and magnetic data, and process the magnetic data to obtain M i The corresponding absolute pose data α i , β i ; α i , β i The horizontal rotation angle and vertical rotation angle expressed in Euler angles are

[0008] (4) Store M1, α1, β1, M i , α i , β i ;

[0009] (5) α1, β1, α i , β i Processed into absolute pose data q represented by quaternion i ;

[0010] (6) Determine whether a set number of data with pose labels have been collected. If so, execute the initialization process step (7); otherwise, continue to execute the initialization process step (3) and the following steps;

[0011] (7) Data preprocessing to obtain image pair M ij and relative pose data q ij ;

[0012] (8) Storage M ij and q ij As a training set;

[0013] (9) The data processing unit loads the twin neural network;

[0014] (10) The data processing unit trains the twin neural network using the training set obtained in the initialization process step (8) to obtain a twin neural network model with optimal network parameters;

[0015] (11) Storing the twin neural network model;

[0016] The working process of the system is as follows:

[0017] (1) The PTZ is controlled to drive the camera to rotate, and the camera collects images synchronously;

[0018] (2) The data processing unit obtains the real-time camera absolute position data;

[0019] (3) outputting the real-time absolute camera pose data of the camera through the communication unit;

[0020] (4) Determine whether the camera has completed its rotation and is in a stationary state. If so, execute system working step (5); otherwise, return to execute system working step (1) and subsequent steps;

[0021] (5) Update the absolute position data of the camera according to the magnetic data;

[0022] The process of the data processing unit obtaining the real-time camera absolute posture data is as follows:

[0023] (1) The acquisition camera acquires the initial image M a , and crop M a To the set size;

[0024] (2) Acquisition termination image M a , and crop M b To the set size;

[0025] (3) M a and M b Paired into image pairs M ab ;

[0026] (4) Align the image to M ab Feed into the twin neural network model;

[0027] (5) Obtain the camera relative pose data q represented by the quaternion of the image pair ab ;

[0028] (6) The obtained q ab Camera relative pose data α converted to Euler angle form ab , β ab ; α ab is the relative horizontal rotation angle in Euler angle form, β ab The relative vertical rotation angle is in Euler angle form;

[0029] (7) Convert the camera relative pose data into real-time camera absolute pose data.

[0030] 1. The camera posture perception system further includes: during the initialization process, the data processing unit pre-processes the data collected by the camera, and the pre-processing process is as follows:

[0031] (1) Acquire image M i , M i Crop to the set size;

[0032] (2) To M i The corresponding absolute pose data α expressed in Euler angles iand β i Converted to absolute pose data q represented by quaternion i =s i +x i a+y i b+z i c, where a, b, and c represent the imaginary part of the quaternion, and s i Represents the real part coefficient of the quaternion, x i 、y i 、z i Represents the coefficient of the imaginary part of the quaternion;

[0033] (3) Collect images M at intervals of n frames j , M j Crop to the set size;

[0034] (4) To M j The corresponding absolute pose data α expressed in Euler angles j and β j Converted to absolute pose data q represented by quaternion j =s j +x j a+y j b+z j c;

[0035] (5) M i and M j Paired into image pairs M ij and use quaternions to represent their relative pose data in M i The absolute pose data q i The inverse operation of

[0036] 2. The camera posture perception system further includes: the camera and pan-tilt head include an integrated camera with a built-in pan-tilt head.

[0037] 3. The camera posture perception system further includes: the focal length of the camera remains unchanged during the operation of the system. If the focal length of the camera changes, the system needs to be reinitialized.

[0038] The deep learning-based camera posture perception system utilizes a twin neural network model to solve the real-time posture perception problem for rotatable cameras. Only the image information from the video surveillance can be used to obtain real-time camera posture information. The system features strong real-time performance, accurate data, and easy installation and implementation. It is widely applicable in public security, factories, transportation, mining, and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of an implementation example of a deep learning-based camera pose perception system.

[0040] Figure 2 An example diagram of the initialization process of a camera posture perception system based on deep learning.

[0041] Figure 3 Schematic diagram of the workflow of the camera posture perception system based on deep learning.

[0042] Figure 4 Schematic diagram of the process of the data processing unit obtaining the real-time camera absolute posture data.

[0043] Figure 5 Schematic diagram of the data processing unit's preprocessing process for data collected by the camera. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the specific embodiments of the present invention and the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application. Figure 1 Schematic diagram of an example implementation of a deep learning-based camera pose perception system.

[0045] 1. A magnetic sensor (101) is used to obtain static magnetic data of the camera and is rigidly connected to the camera body. A GY-511LSM303DLHC magnetic sensor module can be used. The magnetic sensor is connected to the data processing unit via an I2C communication interface to transmit the magnetic data to the data processing unit.

[0046] 2. Data processing unit (102), responsible for receiving and processing magnetic data and video image data, and embedding the twin neural network model. NVIDIA's Jetson nano module with embedded GPU can be used.

[0047] 3. The camera (103) is responsible for collecting images in the format of environmental video streams and sending the collected images to the data processing unit through a communication interface, which can be a USB communication interface.

[0048] 4. The pan-tilt head (104) is responsible for driving the camera (103) to rotate so that the camera can capture a larger image range. The rotation direction includes horizontal and vertical directions, and the pan-tilt head control data is received through the communication unit.

[0049] 5. The communication unit (105) is responsible for sending video image data and camera posture data, and receiving camera setting data and pan / tilt control data. It uses a wired network communication interface, and may also include but is not limited to wireless communication interfaces such as Wi-Fi, Bluetooth, 2G, 3G, 4G, 5G mobile communications, WSN, and wired communication interfaces such as industrial field buses.

[0050] 6. The auxiliary circuit (106) is responsible for providing the required power to the magnetic sensor, data processing unit, camera, pan / tilt head, and communication unit respectively.

[0051] Figure 2 An example diagram of the initialization process of a camera posture perception system based on deep learning. The steps include:

[0052] 1. (201) Capturing image M1 through a camera;

[0053] 2. (202) Collect magnetic data through a magnetic sensor, process the magnetic data to determine the absolute position data α1 and β1 corresponding to M1; α1 and β1 are the horizontal rotation angle and vertical rotation angle expressed in Euler angles, respectively;

[0054] 3. (203) The PTZ drives the camera to rotate and synchronously capture the image M i and magnetic data, and process the magnetic data to obtain M i The corresponding absolute pose data α i , β i ; α i , β i The horizontal rotation angle and vertical rotation angle expressed in Euler angles are

[0055] 4.(204)Store M1, α1, β1, M i , α i , β i ;

[0056] 5. (205) α1, β1, α i , β i Processed into absolute pose data q1, q represented by quaternion i , where q i =s i +x i a+y i b+z i c, a, b, c represent the imaginary part of the quaternion, s i Represents the real part coefficient of the quaternion, x i 、y i 、z i The calculation formula for the imaginary coefficient of the quaternion is:

[0057] 6. (206) Determine whether a set number of data with pose labels have been collected. If so, execute the initialization process step (7). Otherwise, continue to execute the initialization process step (3) and the following steps.

[0058] 7.(207) Data preprocessing to obtain image pair M ij and relative pose data q ij ;

[0059] 8.(208)Storage M ij and q ij As a training set;

[0060] 9. (209) The data processing unit loads the twin neural network;

[0061] 10. (210) The data processing unit trains the twin neural network using the training set obtained in the initialization process step (8) to obtain a twin neural network model with optimal network parameters;

[0062] 11.(211)Store the twin neural network model.

[0063] Figure 3 Schematic diagram of the workflow of the camera posture perception system based on deep learning. The steps include:

[0064] 1. (301) The pan / tilt is controlled to drive the camera to rotate, and the camera collects images synchronously;

[0065] 2. (302) The data processing unit obtains real-time camera pose data;

[0066] 3. (303) Outputting the real-time absolute position data of the camera through the communication unit;

[0067] 4. (304) Determine whether the camera has completed its rotation and is in a stationary state. If so, execute step (305); otherwise, return to step (301);

[0068] 5. (305) Update the camera absolute position data based on the magnetic data.

[0069] Figure 4 A schematic diagram of the process of obtaining real-time camera absolute pose data for the data processing unit, including the following steps:

[0070] 1. (401) The acquisition camera acquires the initial image M a , and crop M a To the set size;

[0071] 2. (402) Acquisition termination image M a , and crop Mb To the set size;

[0072] 3.(403)M a and M b Paired into image pairs M ab ;

[0073] 4. (404) Match the image to M ab Feed into the twin neural network model;

[0074] 5. (405) Obtain the camera relative pose data q represented by the quaternion of the image pair ab ;

[0075] 6.(406) The obtained q ab Camera relative pose data α converted to Euler angle form ab , β ab ; α ab is the relative horizontal rotation angle in Euler angle form, β ab is the relative vertical rotation angle in Euler angle form, where q ab =s ab +x ab a+y ab b+z ab c, which is calculated as follows:

[0076]

[0077] β ab =arcsin[2(s ab y ab -x ab z ab )];

[0078] 7. (407) Convert the camera relative pose data into real-time camera absolute pose data.

[0079] Figure 5 This is a schematic diagram of the process flow for preprocessing the data collected by the camera by the data processing unit, the steps include:

[0080] 1. (501) Collect image M i , M i Crop to the set size, using center cropping

[0081] 2.(502) To M i The corresponding absolute pose data α expressed in Euler angles i and β i Converted to absolute pose data q represented by quaternion i =s i +x i a+yi b+z i c, where a, b, and c represent the imaginary part of the quaternion, and s i Represents the real part coefficient of the quaternion, x i 、y i 、z i Represents the coefficient of the imaginary part of the quaternion;

[0082] 3. (503) Collect images M at intervals of frame number n j , M j Crop to the set size;

[0083] 4.(504) To M j The corresponding absolute pose data α expressed in Euler angles j and β j Converted to absolute pose data q represented by quaternion j =s j +x j a+y j b+z j c;

[0084] 5.(505)M i and M j Paired into image pairs M ij and use quaternions to represent their relative pose data in M i The absolute pose data q i The inverse operation of

[0085]

[0086] In three-dimensional case, rotation is represented by unit quaternion, i.e. Can get

[0087]

[0088]

Claims

1. A camera posture perception system based on deep learning, characterized by: The camera posture perception system comprises: the camera posture perception system comprises: an auxiliary circuit, a magnetic sensor, a camera, a pan-tilt head, a data processing unit and a communication unit; the camera is fixedly mounted on the rotating pan-tilt head for collecting images; the magnetic sensor is used to determine the absolute posture data of the camera when it is stationary; the auxiliary circuit is used to provide power to each part; the data processing unit is used to process the images collected by the camera and the magnetic data collected by the magnetic sensor to obtain real-time absolute posture data of the camera; the pan-tilt head is used to drive the camera to rotate; the communication unit is responsible for sending video image data and camera posture data, and receiving camera setting data and pan-tilt head control data; The camera attitude perception system needs to be initialized before use. The initialization process is as follows: (1) Capturing image M1 through a camera; (2) Magnetic data is collected by a magnetic sensor, and the magnetic data is processed to determine the absolute position data α1 and β1 corresponding to M1; α1 and β1 are the horizontal rotation angle and vertical rotation angle expressed in Euler angles, respectively; (3) The PTZ drives the camera to rotate and synchronously collect image M i and magnetic data, and process the magnetic data to obtain M i The corresponding absolute pose data α i , β i ; α i , β i are the horizontal rotation angle and vertical rotation angle expressed in Euler angles respectively; (4) store M1、a1、b1、M i ,a i 、b i ; (5) α1, β1, α i , β i Processed into absolute pose data q represented by quaternion i ; (6) Determine whether a set number of data with pose labels have been collected. If so, execute the initialization process step (7); otherwise, continue to execute the initialization process step (3) and the following steps; (7) Data preprocessing to obtain image pair M ij and relative pose data q ij ; (8) Storage M ij and q ij As a training set; (9) The data processing unit loads the twin neural network; (10) The data processing unit trains the twin neural network using the training set obtained in the initialization process step (8) to obtain a twin neural network model with optimal network parameters; (11) Storing the twin neural network model; The working process of the system is as follows: (1) The PTZ is controlled to drive the camera to rotate, and the camera collects images synchronously; (2) The data processing unit obtains the real-time camera absolute position data; (3) outputting the real-time absolute camera pose data of the camera through the communication unit; (4) Determine whether the camera has completed its rotation and is in a stationary state. If so, execute system working step (5); otherwise, return to execute system working step (1) and subsequent steps; (5) Update the absolute position data of the camera according to the magnetic data; The process of the data processing unit obtaining the real-time camera absolute posture data is as follows: (1) The acquisition camera acquires the initial image M a , and crop M a To the set size; (2) Acquisition termination image M a , and crop M b To the set size; (3) M a and M b Paired into image pairs M ab ; (4) Align the image to M ab Feed into the twin neural network model; (5) Obtain the camera relative pose data q represented by the quaternion of the image pair ab ; (6) The obtained q ab Camera relative pose data α converted to Euler angle form ab , β ab ; α ab is the relative horizontal rotation angle in Euler angle form, β ab The relative vertical rotation angle is in Euler angle form; (7) Convert the camera relative pose data into real-time camera absolute pose data.

2. The camera posture perception system according to claim 1, wherein: During the initialization process, the data processing unit pre-processes the data collected by the camera, and the pre-processing process is as follows: (1) Acquire image M i , M i Crop to the set size; (2) To M i The corresponding absolute pose data α expressed in Euler angles i and β i Converted to absolute pose data q represented by quaternion i =s i +x i a+y i b+z i c, where a, b, and c represent the imaginary part of the quaternion, and s i Represents the real part coefficient of the quaternion, x i 、y i 、z i Represents the coefficient of the imaginary part of the quaternion; (3) Collect images M at intervals of n frames j , M j Crop to the set size; (4) To M j The corresponding absolute pose data α expressed in Euler angles j and β j Converted to absolute pose data q represented by quaternion j =s j +x j a+y j b+z j c; (5) M i and M j Paired into image pairs M ij and use quaternions to represent their relative pose data in M i The absolute pose data q i The inverse operation of 3. The camera posture perception system according to claim 1, wherein: The camera and the pan / tilt head include an integrated camera with a built-in pan / tilt head.

4. The camera posture perception system according to claim 1, wherein: The focal length of the camera remains unchanged during the operation of the system. If the focal length of the camera changes, the system needs to be reinitialized.

Images