Automatic tracking method and tracking device for high-altitude operation personnel

Abstract

The invention provides an automatic tracking method and tracking device for high-altitude operation personnel. The tracking method comprises the following steps that video information of the operation personnel is collected through a zoom lens; obtaining an image filling target sequence with the operator based on the video information; predicting a motion track of the operator based on a state transition relation of postures of the operator between adjacent frames of images in the target sequence; and the deflection angle of the holder is controlled based on the predicted motion trail, and automatic tracking of the operator is realized. The operator automatic tracking method provided by the invention considers the influence of the attitude of the operator on the trajectory movement direction when the operator moves in a three-dimensional space, can realize high-dimensional information solution and trajectory prediction through a two-dimensional image, avoids the situation that a large amount of operation is needed, accelerates the tracking response speed, and improves the tracking accuracy. The requirement for real-time and accurate monitoring of high-altitude operation can be met, and the safety of operators can be effectively guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of safety equipment for aerial work of electric power equipment, and in particular to an automatic tracking method and a tracking device for aerial work personnel. Background Art

[0002] In the power industry, the maintenance and inspection of power transmission and transformation lines often require personnel to work at heights. For example, during the replacement of high-voltage tower components and the maintenance and installation of pole equipment, construction personnel are required to climb and ascend on power facilities. During high-altitude operations, irregular or illegal behaviors of operators can easily lead to operational safety accidents and even cause loss of life and property.

[0003] In order to improve the safety of operations, it is usually necessary to monitor the operating behavior and operating status of operators. Existing monitoring methods, on the one hand, rely on manual video monitoring, and detection personnel make judgments and decisions by adjusting real-time video images. This monitoring method is highly dependent on the attention of monitoring personnel; on the other hand, it is based on image video target tracking detection to track operators. However, video target tracking algorithms have made effective progress, but when encountering target occlusion, posture changes, and lighting changes, the tracking algorithm has the problem of long-term drift of the target frame and inability to accurately track the target.

[0004] Therefore, there is a technical problem in the prior art that the existing monitoring method for operators cannot accurately track the target and cannot meet the safety requirements of high-altitude operations. Summary of the invention

[0005] The main purpose of the present invention is to provide an automatic tracking method and tracking device for high-altitude workers, so as to solve the technical problem that the existing monitoring method for workers in the prior art cannot accurately track the target and cannot meet the safety requirements of high-altitude operations.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present invention, there is provided a method for collecting video information of a worker through a zoom lens; obtaining images containing the worker based on the video information to fill a target sequence; predicting a motion trajectory of the worker based on a state transition relationship of the worker's posture between adjacent frame images in the target sequence; and controlling a pan-tilt deflection angle based on the predicted motion trajectory to achieve automatic tracking of the worker.

[0007] In some embodiments, collecting the video information of the operator includes collecting the three-dimensional coordinates of the operator through a zoom lens; obtaining adjustment information of the zoom lens based on the three-dimensional coordinates; adjusting the zoom lens based on the adjustment information, and collecting the video information of the operator.

[0008] In some embodiments, obtaining the adjustment information of the zoom lens based on the three-dimensional coordinates includes: adjusting the position of the gimbal according to the three-dimensional coordinates so that the operator is in the middle position of the image captured by the zoom lens; adjusting the focal length of the zoom lens so that the image proportion of the operator in the image reaches a preset value.

[0009] In some embodiments, the method of filling the target sequence with images containing the worker based on the video information includes: obtaining corrected video information based on the video information, wherein the correction method is to eliminate images containing no worker in the video data captured by the zoom lens by using frame drop processing; and filling and updating the target sequence.

[0010] In some embodiments, the method of controlling the deflection angle of the pan / tilt head based on the predicted motion trajectory to achieve automatic tracking of the operator includes:

[0011] Set the coordinate system for the pixel points in the image and set the target point P c The coordinates of point P are transformed into the pixel coordinate system. c The coordinates of the corresponding pixel point in the pixel coordinate system are P p (u, v, 1) T , and set the width and height of a single pixel corresponding to the image to dx and dy respectively, then P c and P p The coordinate transformation between is shown as follows:

[0012]

[0013] Through the internal reference coordinate matrix, the tan value of the deflection angle of the center point of the operator to be tracked relative to the yaw axis and pitch axis of the gimbal is calculated as shown in the following formula:

[0014]

[0015] The corresponding deflection angle is calculated according to the tan value of the deflection angle, and the yaw-axis motor and pitch-axis motor of the gimbal are controlled to rotate through the corresponding deflection angle according to the deflection angle, thereby driving the zoom lens to track the operator.

[0016] In some implementations, the predicting the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence includes:

[0017] Starting from the second frame image of the target sequence, the posture of the operator in the current frame image is calculated according to the Pnp algorithm, and its posture is decomposed into the rotation matrix R t and the translation vector T t ;

[0018] Let the rotation matrix of the operator's posture in the previous frame of the current frame be R t-1 , the translation vector is T t-1 At the same time, let the state transfer matrix between the homogeneous transformation matrix of the operator's posture in the previous frame image and the current frame image be A t ;

[0019] Assume that the homogeneous matrix of the operator's posture in the current frame image is H, and the homogeneous matrix of the operator's posture in the next frame image to be predicted is H', the rotation matrix is ​​R', and the translation vector is T', then the homogeneous matrix H' is calculated by the following formula:

[0020] H′=H·A t ,

[0021] Among them, the homogeneous matrix H′ corresponds to the center point P of the operator in the coordinate system of the predicted next frame image i ; Let point P i The coordinates of (X i , Y i , 1) T , click P i The component on the z-axis of the camera coordinate system is Z c , let the internal parameter coordinate matrix of the zoom lens be K, let the external parameter coordinate matrix of the zoom lens be E; then after obtaining the homogeneous matrix H′, back-project the translation vector to obtain point P i The coordinates are:

[0022]

[0023] Suppose in the camera coordinate system, the next point P of the tracked operator is c The coordinates of (X c , Y c , Z c ) T , where X c , Y c , Z c Point P c Coordinates on the X-axis, Y-axis, and Z-axis; point P c With point P i Correspondingly, the coordinate transformation of the two is expressed as:

[0024] Where f is the focal length of the zoom lens.

[0025] In some embodiments, filling and updating the target sequence includes: when the target sequence capacity is not full, when there are workers in subsequent images, the subsequent images with workers are added to the target sequence in order; when there are no workers in subsequent images, the image frames without workers are discarded, and the counter value of the image is read and added by 1. When the counter value accumulates to the maximum capacity of the target sequence but there are no workers in all subsequent images, the target sequence is deleted and refilled.

[0026] In some embodiments, before collecting video information of the operating personnel through the zoom lens, the method also includes calibrating the zoom lens, including the following steps: first, a black and white chessboard of known size is pointed toward the zoom lens as a calibration plate, the zoom lens is rotated by the pan / tilt, and multiple frames of images of the black and white chessboard are captured when the zoom lens presents different angles relative to the black and white chessboard; then, feature points in the captured images of the black and white chessboard are detected, and the physical coordinate system of the corner points of the calibration plate is calculated based on the known size of the chessboard and the origin of the world coordinate system; finally, based on the physical coordinate system of the corner points of the calibration plate, the intrinsic coordinate matrix and distortion coefficient of the zoom lens are solved, and then the LM algorithm is used to optimize the intrinsic coordinate matrix and distortion coefficient respectively, so as to obtain the zoom lens coordinate system, that is, the camera coordinate system.

[0027] According to another aspect of the present invention, the present invention also provides an automatic tracking device for high-altitude workers, including: an acquisition module, used to acquire video information of workers through a zoom lens; a filling module, which obtains images of the workers based on the video information to fill a target sequence; a prediction module, which predicts the motion trajectory of the workers based on the state transition relationship of the workers' posture between adjacent frame images in the target sequence; and a control module, which controls the deflection angle of the pan / tilt head based on the predicted motion trajectory to achieve automatic tracking of the workers.

[0028] According to another aspect of the present invention, the present invention further provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the aforementioned methods for automatically tracking aerial workers.

[0029] According to another aspect of the present invention, the present invention also provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the aforementioned methods for automatically tracking high-altitude workers when executing the computer program on the memory.

[0030] By applying the technical solution of the present invention, video information of the operator is collected through a zoom lens; based on the video information, images of the operator are obtained to fill the target sequence; the motion trajectory of the operator is predicted from the state transition relationship of the operator's posture between adjacent frame images in the target sequence; thus, the pan-tilt deflection angle can be controlled based on the predicted motion trajectory to achieve automatic tracking of the operator. The automatic tracking method for operators provided by the present invention takes into account the influence of the operator's posture on the trajectory movement direction when the operator moves in three-dimensional space. The high-dimensional information can be solved and the trajectory prediction can be achieved through two-dimensional images, avoiding the situation where a large amount of calculation is required, speeding up the tracking response speed, and meeting the needs of real-time and accurate monitoring of high-altitude operations, which can effectively ensure the safety of operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0032] Figure 1 A flowchart showing the steps of the method for automatically tracking a person working at height according to the present invention is shown;

[0033] Figure 2 A flowchart showing the steps of collecting video information of operators through a zoom lens according to the present invention is shown;

[0034] Figure 3 A flowchart showing the steps of obtaining adjustment information of a zoom lens based on three-dimensional coordinates according to the present invention is shown;

[0035] Figure 4 A flowchart showing the steps of obtaining an image filling target sequence with a worker based on video information according to the present invention is shown;

[0036] Figure 5 A schematic diagram of the process of filling a target sequence with an image according to the present invention is shown;

[0037] Figure 6 It shows a structural block diagram of an automatic tracking device for high-altitude workers according to the present invention;

[0038] Figure 7 A schematic structural diagram of an electronic device according to the present invention is shown. DETAILED DESCRIPTION

[0039] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but are not intended to limit the present disclosure.

[0040] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0042] These and other characteristics of the present disclosure will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the attached drawings.

[0043] It should also be understood that, although the present disclosure has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present disclosure that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0044] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0045] Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but only to serve as a basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of ways with substantially any suitable detailed structure.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0047] This specification may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0048] The present disclosure is further described below in conjunction with the accompanying drawings and specific embodiments.

[0049] In order to solve the technical problem that the existing monitoring method for workers in the prior art cannot accurately track the target and cannot meet the safety requirements of high-altitude operations, the present invention provides an automatic tracking method and tracking device for high-altitude workers.

[0050] Figure 1 The flowchart of the method for automatically tracking high-altitude workers provided by the present invention is shown as follows: Figure 1 As shown, the method for automatically tracking high-altitude workers includes the following steps:

[0051] S101, collecting video information of operators through a zoom lens.

[0052] In this step, video information of the operator is collected through a zoom lens.

[0053] Specifically, the zoom lens can be set on a pan-tilt head, and the pan-tilt head can be set up at a height of 2m from the ground for real-time monitoring of the working scene. The zoom lens can adopt a zoom camera commonly used in the prior art, so as to obtain clear video information. This is not specifically limited in this application.

[0054] Among them, the gimbal is a dual-axis gimbal, so that more precise posture control can be achieved through the dual-axis gimbal.

[0055] Further, Figure 2 The flowchart of the steps of collecting video information of the operator through the zoom lens described in the present invention is shown. As shown in the figure, collecting video information of the operator through the zoom lens includes the following steps:

[0056] S201, collecting the three-dimensional coordinates of the operator through a zoom lens.

[0057] In this step, the three-dimensional coordinates of the operator are collected through a zoom lens.

[0058] Specifically, after collecting the information of the operator through the zoom lens, the neural network image recognition model constructed based on the image detection algorithm can be used to identify the operator in the video data to obtain the three-dimensional coordinates of the operator.

[0059] Among them, firstly, the image recognition model can be trained using training data consisting of images of workers, so that the image recognition model can recognize the workers in the image and output the three-dimensional coordinates of the workers.

[0060] S202: Obtain adjustment information of the zoom lens based on the three-dimensional coordinates.

[0061] After completing the above step S201, in this step, the adjustment information of the zoom lens is obtained based on the three-dimensional coordinates. Figure 3 A flowchart of the steps of obtaining the adjustment information of the zoom lens based on the three-dimensional coordinates is shown. As shown in the figure, obtaining the adjustment information of the zoom lens based on the three-dimensional coordinates includes the following steps:

[0062] S301, adjusting the position of the pan / tilt head according to the three-dimensional coordinates so that the operator is located in the middle of the image captured by the zoom lens.

[0063] In this step, the position of the pan / tilt head is adjusted according to the three-dimensional coordinates so that the operator is in the middle position of the image captured by the zoom lens.

[0064] Specifically, first, the current attitude parameter information of the gimbal is obtained, and the three-dimensional coordinates of the operator and the current attitude parameter information of the gimbal are calculated through a complementary filtering algorithm to obtain attitude adjustment parameters; the attitude of the gimbal is controlled according to the attitude adjustment parameters so that the operator is displayed in the middle position of the shooting screen.

[0065] The current attitude parameter information of the gimbal can be obtained through sensor detection, and the sensor includes a three-axis gyroscope, a three-axis accelerometer, a magnetometer, etc.

[0066] S302, adjusting the focal length of the zoom lens so that the image proportion of the operator in the image reaches a preset value.

[0067] After completing the above step S301, in this step, the focal length of the zoom lens is adjusted so that the image proportion of the operator in the image reaches a preset value.

[0068] Specifically, the image proportion of the operator in the image frame is calculated according to the current image frame taken by the zoom lens, and the focal length parameters of the zoom lens are obtained; and the appropriate focal length parameters are calculated according to the image proportion value and the focal length parameters, so that the image proportion of the operator in the image reaches a preset image proportion value, thereby obtaining the adjustment information of the zoom lens.

[0069] The preset image ratio value may be between 40% and 60%.

[0070] S203: Adjust the zoom lens based on the adjustment information, and collect video information of the operator.

[0071] After completing the above steps, in this step, the zoom lens is adjusted based on the adjustment information, and video information of the operator is collected.

[0072] Specifically, the focal length of the zoom lens is adjusted based on the adjustment information so that the image proportion of the operator in the image meets the requirement of a preset value, and the monitoring video of the operator is collected in real time through the zoom lens with adjusted focal length, and the operator is detected based on the real-time collected video data to achieve capture of the operator.

[0073] Similarly, the workers in the image sequence of video data can be identified by an image recognition model based on a neural network, that is, the image recognition model is first trained using training data consisting of images of workers, so that the image recognition model can identify the workers in the images; then the image sequence of video data obtained by taking a zoom lens is input into the trained image recognition model to identify the workers.

[0074] S102, obtaining an image filling target sequence in which the operator exists based on the video information.

[0075] After completing the above step S101, in this step, an image filling target sequence in which the operator exists is obtained based on the video information.

[0076] Further, Figure 4 The flowchart of the steps of obtaining the image filling target sequence with workers based on video information according to the present invention is shown. Figure 5 FIG. 4 shows a flow chart of image filling target sequence; Figure 4-Figure 5 As shown, obtaining an image filling target sequence with workers based on video information includes the following steps:

[0077] S401, obtaining corrected video information based on the video information, wherein the correction method is to remove images without operators in the video data captured by the zoom lens by using a frame drop processing method.

[0078] In this step, corrected video information is obtained based on the video information, wherein the correction method is to remove images in which no operator exists in the video data captured by the zoom lens by means of frame dropping processing.

[0079] Specifically, after acquiring the video information, a target sequence is created for the first frame image in which the operator exists, wherein the structural particle of the target sequence may be, for example, a double-ended queue with a maximum capacity.

[0080] Identify the image information, and when there are workers in subsequent images, add the subsequent images with workers to the target sequence in order; when there are no workers in subsequent images, discard the image frame without workers, that is, if the current frame target is empty, it will be eliminated, and the counter value of the read image will be increased by 1. When the counter value accumulates to the maximum capacity of the target sequence, but there are no workers in all subsequent images, delete the target sequence, and continue to create a target sequence for the first frame image with workers.

[0081] S402: Fill and update the target sequence.

[0082] After completing the above step S401, in this step, the target sequence is filled and updated.

[0083] Specifically, as mentioned above, when the target sequence capacity is not full, when there are workers in the subsequent images, the subsequent images with workers will be added to the target sequence in order; when there are no workers in the subsequent images, the image frames without workers will be discarded, that is, if the current frame target is empty, it will be eliminated, and the counter value of the read image will be increased by 1. When the counter value accumulates to the maximum capacity of the target sequence, but there are no workers in all subsequent images, the target sequence will be deleted and a new target sequence will be created.

[0084] After the target sequence is full, when there is a worker in the next frame of the image, the image of the earliest frame with the worker is deleted from the head of the queue in order, and then the image of the next frame with the worker is added to the end of the queue, thereby updating the target sequence.

[0085] S103: predicting the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence.

[0086] After completing the above step S102, in this step, the motion trajectory of the operator is predicted based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence.

[0087] Specifically, according to the feature points of each frame of the target sequence, the posture change of the operator in each frame is calculated, and then the trajectory is predicted according to the state transition relationship of the posture between adjacent frames of the image to obtain the center point of the operator. The specific process is:

[0088] Starting from the second frame image of the target sequence, the posture of the operator in the current frame image is calculated according to the Pnp algorithm, and its posture is decomposed into the rotation matrix R t and the translation vector T t ;

[0089] Let the rotation matrix of the operator's posture in the previous frame of the current frame be R t-1 , the translation vector is T t-1 At the same time, let the state transfer matrix between the homogeneous transformation matrix of the operator's posture in the previous frame image and the current frame image be A t ;

[0090] Among them, A t It can be calculated by the following formula:

[0091]

[0092] Assume that the homogeneous matrix of the operator's posture in the current frame image is H, and the homogeneous matrix of the operator's posture in the next frame image to be predicted is H', the rotation matrix is ​​R', and the translation vector is T', then the homogeneous matrix H' is calculated by the following formula:

[0093] H′=H·A t ,

[0094] Among them, the homogeneous matrix H′ corresponds to the center point P of the operator in the coordinate system of the predicted next frame image i ; Let point P i The coordinates of (X i , Y i , 1) T , click P i The component on the z-axis of the camera coordinate system is Z c , let the internal parameter coordinate matrix of the zoom lens be K, let the external parameter coordinate matrix of the zoom lens be E; then after obtaining the homogeneous matrix H′, back-project the translation vector to obtain point P i The coordinates are:

[0095]

[0096] Suppose in the camera coordinate system, the next point P of the tracked operator is c The coordinates of (X c , Y c , Z c ) T , where X c , Y c , Z c Point P c Coordinates on the X-axis, Y-axis, and Z-axis; point P c With point P i Correspondingly, the coordinate transformation of the two is expressed as:

[0097]

[0098] Where f is the focal length of the zoom lens.

[0099] S104, controlling the deflection angle of the pan / tilt head based on the predicted motion trajectory to achieve automatic tracking of the operator.

[0100] After completing the above step S103, in this step, the pan / tilt deflection angle is controlled based on the predicted motion trajectory to achieve automatic tracking of the operator.

[0101] Specifically, point P c The coordinates are converted into a deflection angle relative to the pan / tilt, and then the pan / tilt is controlled according to the deflection angle to drive the zoom lens to rotate to a corresponding angle to achieve automatic tracking of the operator.

[0102] Set the coordinate system for the pixel points in the image and set the target point P c The coordinates of point P are transformed into the pixel coordinate system. c The coordinates of the corresponding pixel point in the pixel coordinate system are P p (u, v, 1) T , and set the width and height of a single pixel corresponding to the image to dx and dy respectively, then P c and P p The coordinate transformation between is shown as follows:

[0103]

[0104] Through the internal reference coordinate matrix, the tan value of the deflection angle of the center point of the operator to be tracked relative to the yaw axis and pitch axis of the gimbal is calculated as shown in the following formula:

[0105]

[0106] The corresponding deflection angle is calculated according to the tan value of the deflection angle, and the yaw-axis motor and pitch-axis motor of the gimbal are controlled to rotate through the corresponding deflection angle according to the deflection angle, thereby driving the zoom lens to track the operator.

[0107] In another embodiment of the present invention, before collecting the video information of the operator through the zoom lens, the method further includes calibrating the zoom lens.

[0108] Specifically, calibrating the zoom lens includes the following steps:

[0109] First, a black and white chessboard of known size is placed towards the zoom lens as a calibration plate. The zoom lens is rotated by the gimbal to capture multiple frames of images of the black and white chessboard when the zoom lens presents different angles relative to the black and white chessboard.

[0110] Then, the feature points in the captured black and white chessboard image are detected, and the physical coordinate system of the corner points of the calibration plate is calculated based on the known chessboard size and the origin of the world coordinate system;

[0111] Finally, according to the physical coordinate system of the corner points of the calibration plate, the intrinsic coordinate matrix and distortion coefficient of the zoom lens are solved, and then the LM algorithm is used to optimize the intrinsic coordinate matrix and distortion coefficient respectively to obtain the zoom lens coordinate system, that is, the camera coordinate system.

[0112] In order to facilitate image processing and fusion, the coordinates of the work surface can also be calibrated, that is, the three-dimensional coordinate system of the target area can be determined, so that multiple cameras in the work surface can be jointly calibrated. At the same time, multiple cameras are numbered to determine the area where each camera is located on the work surface. Therefore, the wide-angle lens corresponding to the area where the coordinates are located can be determined according to the three-dimensional coordinates of the image taken by the zoom lens, and the images taken by the wide-angle lens and the zoom lens can be spliced ​​for multi-view fusion display.

[0113] The automatic tracking method for high-altitude workers provided by the present invention is applied to collect video information of workers through a zoom lens; based on the video information, images of the workers are obtained to fill the target sequence; the motion trajectory of the workers is predicted from the state transition relationship of the workers' postures between adjacent frame images in the target sequence; thus, the deflection angle of the pan / tilt can be controlled based on the predicted motion trajectory to achieve automatic tracking of the workers. The automatic tracking method for workers provided by the present invention takes into account the influence of the workers' postures on the direction of trajectory movement when they move in three-dimensional space, and can achieve high-dimensional information solution and trajectory prediction through two-dimensional images, avoiding the need for a large number of calculations, speeding up the tracking response speed, and meeting the needs for real-time and accurate monitoring of high-altitude operations, and can effectively ensure the safety of workers.

[0114] In order to better implement the above method, the second aspect of the present disclosure further provides an automatic tracking device for high-altitude workers, which can be integrated on an electronic device.

[0115] For example, Figure 6 As shown, the tracking device 200 may include: a collection module 210, a filling module 220, a prediction module 230 and a control module 240, as follows:

[0116] (1) A collection module 210, which is used to collect video information of the operator through a zoom lens.

[0117] (2) A filling module 220, wherein the filling module 220 is used to obtain an image filling target sequence in which the operator exists based on the video information.

[0118] (3) A prediction module 230, which is used to predict the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence.

[0119] (4) A control module 240, wherein the control module 240 is used to control the deflection angle of the pan / tilt head based on the predicted motion trajectory to achieve automatic tracking of the operator.

[0120] Furthermore, the collecting of the video information of the operator includes collecting the three-dimensional coordinates of the operator through a zoom lens; obtaining adjustment information of the zoom lens based on the three-dimensional coordinates; adjusting the zoom lens based on the adjustment information, and collecting the video information of the operator.

[0121] Furthermore, obtaining adjustment information of the zoom lens based on the three-dimensional coordinates includes: adjusting the position of the gimbal according to the three-dimensional coordinates so that the operator is in the middle position of the image captured by the zoom lens; adjusting the focal length of the zoom lens so that the image proportion of the operator in the image reaches a preset value.

[0122] Furthermore, the method of filling the target sequence with images containing the workers based on the video information includes: obtaining corrected video information based on the video information, wherein the correction method is to eliminate images containing no workers in the video data captured by the zoom lens by using frame drop processing; and filling and updating the target sequence.

[0123] The automatic tracking device for high-altitude workers provided by the present invention is used to collect video information of workers through a zoom lens; based on the video information, images of the workers are obtained to fill the target sequence; the motion trajectory of the workers is predicted from the state transition relationship of the workers' postures between adjacent frame images in the target sequence; thus, the deflection angle of the pan / tilt can be controlled based on the predicted motion trajectory to achieve automatic tracking of the workers. The automatic tracking method for workers provided by the present invention takes into account the influence of the workers' postures on the direction of trajectory movement when they move in three-dimensional space. The high-dimensional information can be solved and the trajectory prediction can be achieved through two-dimensional images, avoiding the situation where a large amount of calculations are required, speeding up the tracking response speed, and meeting the needs for real-time and accurate monitoring of high-altitude operations, which can effectively ensure the safety of workers.

[0124] Example 3

[0125] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0126] To this end, a third embodiment of the present disclosure provides a storage medium, which is a computer-readable medium and stores a computer program. When the computer program is executed by a processor, the method provided by the embodiment of the present disclosure is implemented, including the following steps S11 to S14:

[0127] S11, collecting video information of the operator through a zoom lens;

[0128] S12, obtaining an image filling target sequence containing the operator based on the video information;

[0129] S13, predicting the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence;

[0130] S14, controlling the deflection angle of the pan / tilt head based on the predicted motion trajectory to achieve automatic tracking of the operator.

[0131] Furthermore, when the computer program is executed by a processor, other methods provided by any of the above embodiments of the present disclosure are implemented.

[0132] The automatic tracking method for high-altitude workers provided by the present invention is applied to collect video information of workers through a zoom lens; based on the video information, images of the workers are obtained to fill the target sequence; the motion trajectory of the workers is predicted from the state transition relationship of the workers' postures between adjacent frame images in the target sequence; thus, the deflection angle of the pan / tilt can be controlled based on the predicted motion trajectory to achieve automatic tracking of the workers. The automatic tracking method for workers provided by the present invention takes into account the influence of the workers' postures on the direction of trajectory movement when they move in three-dimensional space, and can achieve high-dimensional information solution and trajectory prediction through two-dimensional images, avoiding the need for a large number of calculations, speeding up the tracking response speed, and meeting the needs for real-time and accurate monitoring of high-altitude operations, and can effectively ensure the safety of workers.

[0133] Example 4

[0134] A fourth embodiment of the present disclosure provides an electronic device, such as Figure 7 As shown, the electronic device at least includes a memory 310 and a processor 320. The memory 310 stores a computer program. The processor 320 implements the method provided by any embodiment of the present disclosure when executing the computer program on the memory 310. Exemplarily, the method executed by the electronic device computer program is as follows:

[0135] S21, collecting video information of the operator through a zoom lens;

[0136] S22, obtaining an image filling target sequence containing the operator based on the video information;

[0137] S23, predicting the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence;

[0138] S24, controlling the deflection angle of the pan / tilt head based on the predicted motion trajectory to achieve automatic tracking of the operator.

[0139] In specific implementation, the acquisition module 210, filling module 220, prediction module 230 and control module 240 are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

[0140] The automatic tracking method for high-altitude workers provided by the present invention is applied to collect video information of workers through a zoom lens; based on the video information, images of the workers are obtained to fill the target sequence; the motion trajectory of the workers is predicted from the state transition relationship of the workers' postures between adjacent frame images in the target sequence; thus, the deflection angle of the pan / tilt can be controlled based on the predicted motion trajectory to achieve automatic tracking of the workers. The automatic tracking method for workers provided by the present invention takes into account the influence of the workers' postures on the direction of trajectory movement when they move in three-dimensional space, and can achieve high-dimensional information solution and trajectory prediction through two-dimensional images, avoiding the need for a large number of calculations, speeding up the tracking response speed, and meeting the needs for real-time and accurate monitoring of high-altitude operations, and can effectively ensure the safety of workers.

[0141] The storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0142] The storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0143] Alternatively, the storage medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0144] Computer program code for performing operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the passenger computer, partially on the passenger computer, as a separate software package, partially on the passenger computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the passenger computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] It should be noted that the storage medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any storage medium other than computer readable storage media that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), or any suitable combination of the above.

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0148] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs).

[0149] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0151] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0152] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are only example forms of implementing the claims.

[0153] Multiple embodiments of the present disclosure are described in detail above, but the present disclosure is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concepts of the present disclosure, and these variations and modifications should all fall within the scope of protection required by the present disclosure.

Claims

1. A method for automatically tracking personnel working at heights, characterized in that: include: Collect video information of operators through zoom lens; Obtaining an image filling target sequence in which the operator exists based on the video information; Predicting the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence; The deflection angle of the pan / tilt is controlled based on the predicted motion trajectory to achieve automatic tracking of the operator.

2. The automatic tracking method for high-altitude workers according to claim 1, characterized in that: The video information of the collected operating personnel includes: Collecting the three-dimensional coordinates of the operator through a zoom lens; Obtaining adjustment information of the zoom lens based on the three-dimensional coordinates; The zoom lens is adjusted based on the adjustment information, and video information of the operator is collected.

3. The automatic tracking method for high-altitude workers according to claim 2, characterized in that: The obtaining adjustment information of the zoom lens based on the three-dimensional coordinates includes: Adjust the position of the pan / tilt head according to the three-dimensional coordinates so that the operator is in the middle of the image captured by the zoom lens; Adjust the focal length of the zoom lens so that the image proportion of the operator in the image reaches the preset value.

4. The automatic tracking method for high-altitude workers according to claim 1, characterized in that: The step of obtaining an image filling target sequence in which the operator exists based on the video information includes: Obtaining corrected video information based on the video information, wherein the correction method is to remove images in which no operator is present in the video data captured by the zoom lens by using a frame drop processing method; Fill and update the target sequence.

5. The automatic tracking method for high-altitude workers according to claim 1, characterized in that: The method of controlling the deflection angle of the pan / tilt head based on the predicted motion trajectory to realize automatic tracking of the operator includes: Set the coordinate system for the pixel points in the image and set the target point P c The coordinates of point P are transformed into the pixel coordinate system. c The coordinates of the corresponding pixel point in the pixel coordinate system are P p (u, v, 1) T , and set the width and height of a single pixel corresponding to the image to dx and dy respectively, then P c and P p The coordinate transformation between is shown as follows: Through the internal reference coordinate matrix, the tan value of the deflection angle of the center point of the operator to be tracked relative to the yaw axis and pitch axis of the gimbal is calculated as shown in the following formula: The corresponding deflection angle is calculated according to the tan value of the deflection angle, and the yaw-axis motor and pitch-axis motor of the gimbal are controlled to rotate through the corresponding deflection angle according to the deflection angle, thereby driving the zoom lens to track the operator.

6. The automatic tracking method for high-altitude workers according to claim 1, characterized in that: The predicting of the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence includes: Starting from the second frame image of the target sequence, the posture of the operator in the current frame image is calculated according to the Pnp algorithm, and its posture is decomposed into the rotation matrix R t and the translation vector T t ; Let the rotation matrix of the operator's posture in the previous frame of the current frame be R t-1 , the translation vector is T t-1 At the same time, let the state transfer matrix between the homogeneous transformation matrix of the operator's posture in the previous frame image and the current frame image be A t ; Assume that the homogeneous matrix of the operator's posture in the current frame image is H, and the homogeneous matrix of the operator's posture in the next frame image to be predicted is H', the rotation matrix is ​​R', and the translation vector is T', then the homogeneous matrix H' is calculated by the following formula: H′H·A t , Among them, the homogeneous matrix H′ corresponds to the center point P of the operator in the coordinate system of the predicted next frame image i ; Let point P i The coordinates of (X i , Y i , 1) T , click P i The component on the Z axis of the camera coordinate system is Z c , let the internal parameter coordinate matrix of the zoom lens be K, let the external parameter coordinate matrix of the zoom lens be E; then after obtaining the homogeneous matrix H′, back-project the translation vector to obtain point P i The coordinates are: Suppose in the camera coordinate system, the next point P of the tracked operator is c The coordinates of (X c , Y c , Z c ) T , where X c , Y c , Z c Point P c Coordinates on the X-axis, Y-axis, and Z-axis; point P c With point P i Correspondingly, the coordinate transformation of the two is expressed as: Where f is the focal length of the zoom lens.

7. The automatic tracking method for high-altitude workers according to claim 4, characterized in that: The filling and updating of the target sequence includes: when the target sequence capacity is not full, when there are workers in subsequent images, the subsequent images with workers are added to the target sequence in order; when there are no workers in subsequent images, the image frames without workers are removed, and the counter value of the image is read and added by 1. When the counter value accumulates to the maximum capacity of the target sequence but there are no workers in all subsequent images, the target sequence is deleted and the target sequence is filled again.

8. An automatic tracking device for high-altitude workers, characterized in that: include: A collection module, used to collect video information of operators through a zoom lens; A filling module, which obtains an image filling target sequence in which the operator exists based on the video information; A prediction module predicts the motion trajectory of the operator based on the state transition relationship of the operator's posture between adjacent frame images in the target sequence; The control module controls the deflection angle of the pan / tilt head based on the predicted motion trajectory to achieve automatic tracking of the operator.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for automatically tracking a person working at height according to any one of claims 1 to 7 are implemented.

10. An electronic device, comprising at least a memory and a processor, wherein a computer program is stored in the memory, wherein: The processor implements the steps of the method for automatically tracking aerial workers according to any one of claims 1 to 7 when executing the computer program on the memory.

Images