Follow-up holder automatic tracking method based on instrument image recognition

Abstract

The invention relates to the technical field of visual tracking and depth measurement, in particular to a follow-up holder automatic tracking method based on instrument image recognition. According to the motion law of the pan-tilt camera, when the target dial plate is shot, the included angle between the shooting depths at the two adjacent shooting positions is obtained to serve as the transformation angle delta phi, and during shooting each time, the pan-tilt angle is adjusted according to the transformation angle delta phi so as to track the target dial plate. According to the follow-up holder automatic tracking method based on instrument image recognition, a traditional trolley position adjusting mode is replaced with a posture adjusting mode, after the trolley reaches the fixed position, the angle of the camera is adjusted through the holder to clearly shoot an instrument panel, and therefore the accuracy of tracking of the instrument panel is improved. In this way, the limitation that too many point positions cannot be set in the petrochemical industry and the like is avoided, the environmental limitation of depth measurement is reduced, and the application range of depth measurement is widened.

Description

Technical Field

[0001] The present application relates to the technical field of visual tracking and depth measurement, and in particular to an automatic tracking method of a follow-up pan / tilt head based on instrument image recognition. Background Art

[0002] With the rapid development and widespread popularity of chemical plants in recent years, preventing the leakage of harmful gases or liquids in chemical plants has become a key issue, which is the key to effectively ensure the safety of employees and surrounding residents and protect property from loss. For the daily inspection of instrument values ​​in a large number of chemical plants, it is still carried out by manual investigation. The judgment of instrument values ​​in this way is greatly affected by subjective factors and the inspection efficiency is low. At present, a computer vision-based method has been developed to realize the automatic recognition technology of instrument values ​​to replace the traditional manual investigation method, greatly improve the recording of instrument values, and avoid various serious performance degradation caused by situations that are not within the normal range.

[0003] Traditional fixed recognition is a single camera installed on a gimbal, and the object is recognized by modifying the position of the vehicle and the horizontal and vertical rotation of the gimbal. The camera position is too fixed. For objects that are difficult to detect, the camera obtains the position of the recognized object and then adjusts the position of the vehicle. This operation is no problem in open places, but in the petrochemical industry, due to explosion-proof requirements, vehicles cannot set a large number of points and lay tracks to meet this requirement, and the accuracy of SLAM navigation still needs to be improved.

[0004] Application Contents

[0005] In order to overcome the defects of the above-mentioned prior art method of first adjusting the position of the car and then adjusting the angle of the gimbal camera for depth measurement, which has many limitations and low efficiency, the present application proposes a method of pre-calculating the transformation angle according to the distance between adjacent shooting positions and pre-adjusting it, thereby improving the efficiency and accuracy of depth measurement.

[0006] The present invention proposes a method for automatic tracking of a follow-up pan-tilt head based on instrument image recognition. According to the movement law of the pan-tilt camera, when shooting the target dial, the angle between the shooting depths of two adjacent shooting positions is obtained as the transformation angle. In each shot, according to the change of angle Adjust the gimbal angle to track the target dial.

[0007] Preferably, firstly, an instrument recognition model is trained to segment the instrument panel from the captured image; the gimbal camera moves along a set track to capture the instrument; the gimbal camera is provided with an idle mode and a tracking mode; the capture period of the gimbal camera in the idle mode is greater than the capture period of the gimbal in the tracking mode;

[0008] The initial working state of the gimbal camera is idle mode, and the instrument recognition model recognizes the captured image of the gimbal camera; when the instrument recognition model is detected in the captured image, the gimbal camera switches to tracking mode; until the instrument tracking is completed, the gimbal camera switches back to idle mode;

[0009] In tracking mode, the gimbal camera first adjusts the field of view and calculates the minimum shooting depth in the direction of the gimbal camera's movement and the angle between the shooting depth and the minimum shooting depth when switching to tracking mode; then the gimbal camera is driven to move, and the angle difference between the shooting depths is calculated as the transformation angle based on the distance between two adjacent shooting locations. And every time you shoot, change the angle Adjust the gimbal angle to track the target dial.

[0010] Preferably, in the tracking mode, the gimbal camera first shoots the target instrument at two different shooting positions, and records the shooting depth and the distance between the two shooting positions, and then calculates the minimum shooting depth L;

[0011]

[0012] Wherein, D1 and D2 are the shooting depths obtained from the two shots, respectively, and Δx is the distance between the two shooting positions.

[0013] Preferably, in tracking mode, the change angle between two adjacent shots Calculated according to the following formula:

[0014]

[0015] Among them, λ is the angle between the current shooting position and the minimum shooting depth, is the angle between the next shooting position and the minimum shooting depth; x is the distance between the tracking starting point and the shooting position where the minimum shooting depth is located; d is the distance between the current shooting position and the next shooting position; and x' is the distance between the next shooting position and the tracking starting point.

[0016] Preferably, in tracking mode, the gimbal camera first shoots the target dial at different positions and calculates the minimum shooting depth L; then returns to the tracking starting point and moves toward the tracking end point at a constant speed, and shoots the target dial according to the set frequency until it reaches the tracking end point and switches to the idle state.

[0017] Preferably, in tracking mode, the target of the gimbal camera to adjust the field of view is: the target dial is in the center of the field of view, and the area ratio of the instrument panel identification frame marked by the instrument identification model in the field of view is greater than or equal to 1 / 3 and less than 1.

[0018] Preferably, the instrument recognition model uses a YOLOv5 model trained to convergence on the data set {captured image; dial recognition frame, pointer reading}.

[0019] Preferably, the shooting position when the gimbal camera switches to the tracking mode is used as the tracking starting point, and the point where the minimum shooting depth is located at the tracking starting point is used as the tracking end point; when the gimbal camera reaches or passes the tracking end point, the gimbal camera switches from the tracking mode to the idle mode.

[0020] Preferably, the images captured by the gimbal camera are filtered to select qualified images; then the qualified images corresponding to each instrument are selected, and the average value of the dial index in the qualified image corresponding to the instrument is used as the reading of the instrument.

[0021] Preferably, an identification mark is provided on the instrument panel, and the qualified images are classified according to the identification mark, and the mean of the dial index in the qualified images in the class is calculated as the reading of the instrument panel corresponding to the class.

[0022] The advantages of this application are:

[0023] (1) The present application proposes a method for automatic tracking of a follow-up gimbal based on instrument image recognition. The method uses a posture adjustment method instead of the traditional method of adjusting the position of the trolley. After the trolley reaches a fixed position, the gimbal adjusts the camera angle to take a clear picture of the instrument panel. This avoids the limitation of not being able to set too many points in the petrochemical industry, reduces the environmental limitations of depth measurement, and increases the application scope of depth measurement.

[0024] (2) This application uses a depth camera to predict the rotation angle in advance and realizes follow-up pan-tilt shooting, which reduces the requirements for point positions and camera motion trajectories in the detection environment. It can greatly change the limitations of dial detection in special environments such as petrochemical industries and improve recognition success rate and accuracy. This application can flexibly shoot instruments at different positions and angles in the environment. It is expected to accurately capture clear dashboard images directly from complex industrial environments and identify specific instrument values.

[0025] (3) This application uses deep learning methods for image recognition, which can hand over complex calculation processes to neural networks for learning. By modifying the image, the objects that need to be marked are easier to mark, while eliminating the surrounding noise and retaining the outline of the target that needs to be marked. Without the need for cameras with too high pixels, human intervention is reduced, making it easier to patrol and inspect the dashboards in the factory.

[0026] (4) This application combines camera motion monitoring and angle recording to adjust the camera posture at the shooting position, and then the camera adaptively adjusts the focal length and takes pictures to complete the visual inspection. This application replaces the traditional manual adjustment or setting of fixed inspection methods, and only uses an end-to-end approach to completely get rid of human involvement in the inspection process.

[0027] (5) The present application can photograph the instrument panel at multiple points through the follow-up tracking mode, which can effectively avoid the situation where it is impossible to obtain valid readings due to occlusion, reflection and other unavoidable situations in factory operations. At the same time, averaging the readings at multiple points can also effectively improve the recognition accuracy.

[0028] (6) After identifying a specific dial, the present application uses an identification mark on the instrument panel, such as a QR code, to read the specific name and parameters of the instrument, which facilitates statistics and also facilitates statisticians to check and organize data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of a method for automatic tracking of a follow-up pan / tilt head based on instrument image recognition;

[0030] Figure 2 Schematic diagram of the minimum depth calculation method;

[0031] Figure 3 Schematic diagram of the transformation angle calculation method. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] This embodiment proposes an automatic tracking method for a follow-up gimbal based on instrument image recognition. First, an instrument recognition model is trained to segment the instrument panel from the captured image. The gimbal camera is provided with an idle mode and a tracking mode. In the idle mode, the shooting period of the gimbal camera is greater than the shooting period of the gimbal in the tracking mode. In both the idle mode and the tracking mode, the gimbal moves along a set track.

[0034] The initial working state of the gimbal camera is idle mode. The instrument recognition model recognizes the captured image of the gimbal camera to determine whether the instrument recognition model exists in the captured image. When the instrument recognition model is detected in the captured image, the gimbal camera switches to tracking mode. After the instrument tracking is completed, the gimbal camera switches back to idle mode. The conditions for completing instrument tracking are as follows: the gimbal camera leaves the tracking area. The shooting position when the gimbal camera switches to the tracking mode is taken as the tracking starting point, and the point where the tracking starting point is located with respect to the minimum shooting depth is taken as the tracking end point. The tracking area is the area from the tracking starting point to the tracking end point.

[0035] In tracking mode, the gimbal camera first adjusts the field of view so that the target dial is in the center of the field of view, and the instrument panel identification frame accounts for 1 / 3 to 1 / 2 of the field of view; then it tracks and shoots the target dial, and calculates the angle difference between the shooting depths at two adjacent shooting locations as the transformation angle. And every time you shoot, according to the change of angle Adjust the gimbal angle to track the target dial.

[0036] Reference Figure 1 , Figure 2 In tracking mode, the method of tracking the target dial specifically includes the following steps:

[0037] S11. When the gimbal camera switches to tracking mode, the current position is recorded as the tracking starting point, and the gimbal angle is adjusted so that the target dial is in the center of the field of view, and the area ratio of the instrument panel identification frame in the field of view reaches 1 / 3 to 1 / 2.

[0038] S12, driving the gimbal to move, and taking two images during the movement, recording the shooting depths of the two shots and calculating the minimum shooting depth L;

[0039]

[0040] Wherein, L is the calculated minimum depth of the target dial from the forward position, D1 and D2 are the shooting depths obtained from the two shots, respectively; Δx is the moving distance of the robot between the two shots, that is, the distance between the two shooting positions.

[0041] Specifically, in this step, the shooting angle needs to be adjusted when the gimbal is shooting, so that the target dial is in the center of the field of view, and the instrument panel recognition frame accounts for 1 / 3 to 1 / 2 of the area of ​​the field of view. The target dial recognition frame refers to the recognition frame marked by the instrument recognition model on the captured image.

[0042] S13, move the gimbal and take pictures, obtain the moving distance between two consecutive pictures, and calculate the transformation angle between two consecutive pictures Based on the last shooting angle and changing angle Adjust the gimbal angle so that when the gimbal reaches the next shooting location, it is adjusted to the corresponding shooting angle for the next shooting.

[0043] The moving distance between two adjacent shots is calculated based on the robot's moving speed and the shooting frequency; in specific implementation, the robot can be set to move at a constant speed.

[0044] In step S12, the shooting depth D1 corresponds to the tracking starting point. In step S12, the distance x between the tracking starting point and the shooting position corresponding to the minimum shooting depth is also calculated:

[0045]

[0046] The calculation of the change angle in step S13 also needs to take into account the change in the shooting depth. S13 specifically includes the following sub-steps.

[0047] S131, setting the shooting frequency and calculating the moving distance d between two consecutive shootings; returning the gimbal to the tracking starting point, adjusting the gimbal angle to shoot the target dial, so that the target dial is in the center of the field of view, and the area ratio of the instrument panel identification frame in the field of view reaches 1 / 3 to 1 / 2.

[0048] S132: Move the gimbal and take photos, and calculate the change angle between two consecutive photos. according to Adjust the gimbal angle to track the target dial; during the movement, monitor the gimbal moving distance x' in real time;

[0049]

[0050] λ is the angle between the current shooting position and the minimum shooting depth. Figure 3 .

[0051] The meter reading method in this embodiment includes the following steps.

[0052] S1. First, an identification mark is set on each instrument panel. The identification mark can be set as a code, a QR code, etc.; then the instrument panel is tracked by using the above-mentioned instrument image recognition-based follow-up pan / tilt automatic tracking method.

[0053] S2. Filter the images captured by the gimbal camera to select qualified images. The filtering condition can be specifically set as a confidence threshold, and the images captured with a confidence greater than or equal to the set confidence threshold are selected as qualified images.

[0054] S3. Classify the qualified images according to the identification marks, and calculate the mean of the dial index in the qualified images in the class as the reading of the instrument panel corresponding to the class.

[0055] Specifically, the dial index on the image can be identified and output through the instrument recognition model.

[0056] The instrument recognition model can specifically adopt the YOLOv5 model, which is trained on the data set {shot image; dial recognition frame, pointer reading}.

[0057] The images in the dataset include the original images directly captured by the PTZ camera and the enhanced images obtained by enhancing the original images; image enhancement includes: size transformation, horizontal flip, 90-degree rotation, 270-degree rotation, integer translation and histogram equalization operations, etc. The dial recognition frame in the dataset can be manually marked, and the pointer reading can be manually marked or recognized by a computer with dial reading function.

[0058] Of course, for those skilled in the art, the present application is not limited to the details of the above exemplary embodiments, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims. Any figure mark in the claims should not be regarded as limiting the claim involved.

[0059] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0060] The technologies, shapes, and structural parts not described in detail in this application are all well-known technologies.

Claims

1. A method for automatic tracking of a follow-up pan / tilt head based on instrument image recognition, characterized in that: According to the movement law of the gimbal camera, when shooting the target dial, the angle between the shooting depths of two adjacent shooting positions is obtained as the transformation angle In each shot, according to the change of angle Adjust the gimbal angle to track the target dial.

2. The automatic tracking method of a follow-up pan / tilt head based on instrument image recognition according to claim 1, characterized in that: First, an instrument recognition model is trained to segment the instrument panel from the captured image; the gimbal camera moves along a set track to capture the instrument; the gimbal camera is provided with an idle mode and a tracking mode; the capture period of the gimbal camera in the idle mode is greater than the capture period of the gimbal in the tracking mode; The initial working state of the gimbal camera is idle mode, and the instrument recognition model recognizes the image taken by the gimbal camera; when the instrument recognition model is detected in the captured image, the gimbal camera switches to tracking mode; After the instrument tracking is completed, the gimbal camera switches back to idle mode; In tracking mode, the gimbal camera first adjusts the field of view and calculates the minimum shooting depth in the direction of the gimbal camera's movement and the angle between the shooting depth and the minimum shooting depth when switching to tracking mode; then the gimbal camera is driven to move, and the angle difference between the shooting depths is calculated as the transformation angle based on the distance between two adjacent shooting locations. And every time you shoot, according to the change of angle Adjust the gimbal angle to track the target dial.

3. The automatic tracking method of a follow-up pan / tilt platform based on instrument image recognition according to claim 2, characterized in that: In tracking mode, the gimbal camera first shoots the target instrument at two different shooting positions, records the shooting depth and the distance between the two shooting positions, and then calculates the minimum shooting depth L; Wherein, D1 and D2 are the shooting depths obtained from the two shots, respectively, and Δx is the distance between the two shooting positions.

4. The automatic tracking method of a follow-up pan / tilt head based on instrument image recognition according to claim 3, characterized in that: In tracking mode, the change angle between two consecutive shots Calculated according to the following formula: Among them, λ is the angle between the current shooting position and the minimum shooting depth, is the angle between the next shooting position and the minimum shooting depth; x is the distance between the tracking starting point and the shooting position where the minimum shooting depth is located; d is the distance between the current shooting position and the next shooting position; and x' is the distance between the next shooting position and the tracking starting point.

5. The automatic tracking method of a follow-up pan / tilt platform based on instrument image recognition according to claim 4, characterized in that: In tracking mode, the gimbal camera first photographs the target dial at different positions and calculates the minimum photographing depth L; Then it returns to the starting point of tracking and moves toward the end point of tracking at a constant speed, shooting the target dial according to the set frequency until it reaches the end point of tracking and switches to the idle state.

6. The automatic tracking method of a follow-up pan / tilt platform based on instrument image recognition according to claim 2, characterized in that: In tracking mode, the gimbal camera adjusts the field of view to the following goals: the target dial is in the center of the field of view, and the area ratio of the instrument panel identification frame marked by the instrument identification model in the field of view is greater than or equal to 1 / 3 and less than 1.

7. The automatic tracking method of a follow-up pan / tilt head based on instrument image recognition according to claim 2, characterized in that: The instrument recognition model adopts the dataset {taken images; Dial recognition box, pointer reading} YOLOv5 model trained to convergence.

8. The automatic tracking method of a follow-up pan / tilt platform based on instrument image recognition according to claim 1, characterized in that: The shooting position when the gimbal camera switches to the tracking mode is taken as the tracking start point, and the point of the tracking start point relative to the minimum shooting depth is taken as the tracking end point; when the gimbal camera reaches or passes the tracking end point, the gimbal camera switches from the tracking mode to the idle mode.

9. The automatic tracking method for a follow-up pan / tilt platform based on instrument image recognition according to any one of claims 1 to 8, characterized in that: The images taken by the PTZ camera are filtered to select qualified images; then the qualified images corresponding to each instrument are selected, and the average value of the dial index in the qualified images corresponding to the instrument is used as the reading of the instrument.

10. The automatic tracking method of a follow-up pan / tilt platform based on instrument image recognition according to claim 9, characterized in that: There are identification marks on the instrument panel. The qualified images are classified according to the identification marks, and the mean of the dial index in the qualified images in the class is calculated as the reading of the instrument panel corresponding to the class.

Images