Robot shooting positioning method, device, equipment and medium

Abstract

The invention relates to the technical field of shooting positioning, and discloses a robot shooting positioning method, device and equipment and a medium, and the method comprises the steps: moving a robot to a preset operation position, rotating a holder to a preset initial pose, and obtaining an actual image of an auxiliary fixed point marker, determining a shooting deviation according to the reference image and the actual image of the auxiliary fixed point marker; adjusting a preset initial pose of the holder for multiple times according to the shooting deviation until the shooting deviation does not exist, and obtaining a corrected initial pose of the holder; and rotating the holder from the corrected initial pose according to the preset pose variation, setting the zoom ratio of the image acquisition equipment as a preset multiple, and shooting the target object. Through multiple times of shooting and pose correction, it is ensured that the target object is locked and shot more accurately during high-power zoom shooting, the robot cradle head can locate the target object more quickly and shoot the target object, high-quality inspection data are obtained, the inspection efficiency is improved, and the labor cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of shooting and positioning technology, and in particular to a robot shooting and positioning method, device, equipment and medium. Background Art

[0002] With the improvement of automation and electrification technology, more and more scenes are beginning to use robots to replace manual inspections, such as river inspections, power inspections, building inspections, etc. Most of these scenes use robots to carry rotating pan-tilt heads for shooting. During the actual inspection process of the robot, after the robot reaches the preset working position, it will rotate through the pan-tilt head to enable the image acquisition device to capture the image of the target object to realize the inspection of the target object. Since the target objects to be photographed are mostly far away from the robot itself, the robot pan-tilt camera needs to be magnified several times to capture the target object clearly. In the process of the robot shooting precise objects at a long distance, the shooting distance is too far and the magnification ratio of the captured image is often large, resulting in the shooting deviation of the captured object due to the insufficient repeated positioning accuracy of the robot, resulting in the zoomed-in object often being out of the field of view. Summary of the invention

[0003] In view of this, the present invention provides a robot shooting and positioning method, device, equipment, and medium to solve the problem in the prior art that the target object is not within the field of view due to the long shooting distance and the excessive magnification ratio of the shot image.

[0004] In a first aspect, the present invention provides a robot shooting positioning method, wherein a pan / tilt installed on the robot is used to support an image acquisition device, and the method comprises: Move the robot to the preset working position, rotate the gimbal to the preset initial posture, obtain the actual image of the auxiliary fixed-point marker, and determine the shooting deviation based on the reference image and actual image of the auxiliary fixed-point marker. The reference image is obtained by shooting the auxiliary fixed-point marker during the gimbal calibration process. The preset initial posture of the gimbal is adjusted multiple times according to the shooting deviation until there is no shooting deviation, thereby obtaining a corrected initial posture of the gimbal; According to the preset posture change amount, the gimbal is rotated from the calibrated initial posture, the zoom ratio of the image acquisition device is set to the preset multiple, and the target object is photographed. The preset posture change amount is the posture change amount of the gimbal from shooting the auxiliary fixed-point marker to shooting the target object during the gimbal calibration process.

[0005] The robot shooting and positioning method provided by the present invention corrects the posture of the gimbal by shooting auxiliary fixed-point markers multiple times, ensuring more accurate locking and shooting of the target object during high-power zoom shooting, so that the robot gimbal can locate the target object and shoot more quickly, obtain high-quality inspection data, improve inspection efficiency, and reduce labor costs.

[0006] In an optional implementation, the gimbal calibration process includes: Move the robot to the preset working position, rotate the gimbal to the preset initial posture, and obtain the reference image of the auxiliary fixed-point marker; Adjust the gimbal posture, set the zoom ratio of the image acquisition device to a preset multiple, place the target object within the shooting range of the gimbal, record the target posture of the gimbal, and determine the preset posture change based on the target posture and the preset initial posture.

[0007] The robot shooting and positioning method provided by the present invention calibrates the pan-tilt head at a preset working position in advance to determine the reference image and the change in posture, which is beneficial to compensate for the pan-tilt head posture during subsequent inspections and improves the shooting efficiency of the target object.

[0008] In an optional implementation, determining the shooting deviation according to the reference image and the actual image of the auxiliary fixed-point marker includes: Determine the coordinates of the reference center point of the auxiliary fixed-point marker in the reference image using the pre-trained model; The pre-trained model is used to determine the actual center point coordinates of the auxiliary fixed-point markers in the actual image; The deviation between the reference center point coordinates and the actual center point coordinates is calculated as the shooting deviation.

[0009] The robot shooting positioning method provided by the present invention uses a preset model to determine the reference center point coordinates and the actual center point coordinates, and determines the shooting deviation based on this, finds the shooting deviation in the shooting picture, and is convenient for correcting and compensating the gimbal posture based on this, ensuring that the target object does not exceed the shooting range when the target object is subsequently shot.

[0010] In an optional implementation, determining the reference center point coordinates of the auxiliary fixed-point marker in the reference image using a pre-trained model includes: Acquire multiple sample images containing auxiliary fixed-point markers, and annotate the bounding box of the auxiliary fixed-point marker in each sample image; The preset model is trained using sample images with marked bounding boxes to obtain a pre-trained model; Input the reference image into the pre-trained model, obtain the bounding box of the auxiliary fixed-point marker in the reference image, and determine the vertex coordinates of the bounding box in the reference image based on the image coordinate system; The fiducial center point coordinates are calculated based on the vertex coordinates of the bounding box in the fiducial image.

[0011] In an optional implementation, calculating the coordinates of the reference center point according to the vertex coordinates of the bounding box in the reference image includes: Get the coordinates of the upper left corner and the lower right corner of the bounding box in the reference image; Calculate the average value of the upper left corner vertex coordinates and the lower right corner vertex coordinates as the reference center point coordinates.

[0012] The robot shooting and positioning method provided by the present invention performs bounding box annotation on multiple sample images of auxiliary fixed-point markers, and then trains a preset model. The trained pre-trained model processes the reference image to obtain accurate vertex coordinates of the bounding box, so as to achieve the purpose of quickly determining the left side of the center point and improve the efficiency of gimbal posture correction.

[0013] In an optional implementation, adjusting the preset initial posture of the gimbal multiple times according to the shooting deviation includes: According to the preset conversion relationship between the image coordinate system and the gimbal coordinate system, the shooting deviation is converted from the image coordinate system to the gimbal coordinate system to obtain the gimbal deviation; Adjust the preset initial posture of the gimbal multiple times according to the gimbal deviation.

[0014] The robot shooting positioning method provided by the present invention obtains the gimbal deviation by converting the shooting deviation from the image coordinate system to the gimbal coordinate system, so that the gimbal internal control system can adjust the gimbal posture according to the gimbal deviation, thereby improving the accuracy of the gimbal posture adjustment.

[0015] In an optional implementation, the preset conversion relationship between the image coordinate system and the gimbal coordinate system is:

[0016] Where s represents the scaling factor, represents the coordinates of the target point in the image coordinate system, A represents the intrinsic parameter matrix of the gimbal camera, represents the extrinsic parameter matrix of the gimbal camera, Indicates the coordinate system of the gimbal The coordinates of the corresponding target point.

[0017] The robot shooting positioning method provided by the present invention determines the preset conversion relationship between the image coordinate system and the gimbal coordinate system through the gimbal's own parameters, which is conducive to quickly determining the gimbal deviation and improving the efficiency of gimbal posture correction and shooting target objects.

[0018] In a second aspect, the present invention provides a robot shooting positioning device, wherein a pan / tilt installed on the robot is used to support an image acquisition device, and the device comprises: The shooting deviation determination module is used to move the robot to a preset working position, rotate the gimbal to a preset initial posture, obtain the actual image of the auxiliary fixed-point marker, and determine the shooting deviation based on the reference image and actual image of the auxiliary fixed-point marker. The reference image is obtained by shooting the auxiliary fixed-point marker during the gimbal calibration process. A posture correction module is used to adjust the preset initial posture of the gimbal multiple times according to the shooting deviation until there is no shooting deviation, thereby obtaining a corrected initial posture of the gimbal; The target object shooting module is used to rotate the gimbal from the calibrated initial posture according to the preset posture change, set the zoom ratio of the image acquisition device to a preset multiple, and shoot the target object. The preset posture change amount is the posture change amount of the image acquisition device from shooting the auxiliary fixed-point marker to shooting the target object gimbal during the gimbal calibration process.

[0019] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to execute the method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a schematic flow chart of a robot shooting and positioning method according to an embodiment of the present invention; Figure 2 is a schematic flow chart of another robot shooting and positioning method according to an embodiment of the present invention; Figure 3 is a schematic diagram of the positional relationship between a pixel coordinate system and an imaging plane coordinate system in a robot shooting and positioning method according to an embodiment of the present invention; Figure 4 is a schematic diagram of pinhole imaging in a robot shooting and positioning method according to an embodiment of the present invention; Figure 5 is a schematic diagram of the positional relationship between the gimbal coordinate system and the imaging plane coordinate system in the robot shooting and positioning method according to an embodiment of the present invention; Figure 6 is a structural block diagram of a robot shooting and positioning device according to an embodiment of the present invention; Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] An embodiment of the present invention provides a robot shooting and positioning method, which corrects the posture of the gimbal by shooting auxiliary fixed-point markers multiple times, so as to achieve the effect of the robot gimbal quickly positioning the target object and shooting.

[0025] According to an embodiment of the present invention, an embodiment of a robot shooting and positioning method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] In this embodiment, a robot shooting and positioning method is provided, which can be used in the above-mentioned computer system. Figure 1 is a flow chart of a robot shooting and positioning method according to an embodiment of the present invention, wherein the pan / tilt installed on the robot is used to support an image acquisition device, such as Figure 1 As shown, the process includes the following steps: Step S101, move the robot to a preset working position, rotate the gimbal to a preset initial posture, obtain the actual image of the auxiliary fixed-point marker, and determine the shooting deviation based on the reference image and actual image of the auxiliary fixed-point marker. The reference image is obtained by shooting the auxiliary fixed-point marker during the gimbal calibration process.

[0027] Specifically, during the inspection process, the robot's working position for photographing each target object is fixed. The robot is moved to the preset working position and the gimbal is rotated to the preset initial posture. At this time, due to the deviation of the robot and the gimbal itself, it may not be possible to directly rotate the gimbal to the gimbal posture determined during the calibration process when photographing the target object to obtain the image of the target object. Therefore, it is necessary to calibrate the shooting error of the robot's gimbal with the help of auxiliary fixed-point markers that are fixed relative to the target object. Compared with the target object, the auxiliary fixed-point markers are larger in size and closer to the robot's gimbal. During the inspection process, each time the robot reaches the preset working position and the gimbal rotates to the preset initial posture, the actual image of the auxiliary fixed-point marker is obtained, and the shooting deviation is determined based on the reference image of the auxiliary fixed-point marker and the actual image. The reference image is obtained by photographing the auxiliary fixed-point marker during the gimbal calibration process.

[0028] Step S102, adjusting the preset initial posture of the gimbal for multiple times according to the shooting deviation until there is no shooting deviation, thereby obtaining the corrected initial posture of the gimbal.

[0029] Specifically, the preset initial posture of the gimbal is adjusted according to the shooting deviation. After each adjustment, the shooting deviation between the reference image and the actual image obtained after the adjustment is calculated again. The step of adjusting the preset initial posture of the gimbal according to the shooting deviation is repeated according to the new shooting deviation until there is no shooting deviation or the shooting deviation is 0, and the posture of the gimbal at this time is used as the corrected initial posture. It should be noted that, according to the actual shooting situation, the deviation range of the shooting deviation can also be set. When the shooting deviation exceeds the deviation range, the preset initial posture of the gimbal is adjusted. If the shooting deviation is within the preset deviation range, there is no need to continue to adjust the gimbal posture. This is only an example, but it is not limited to this.

[0030] Step S103, rotate the gimbal from the calibrated initial posture according to the preset posture change amount, set the zoom ratio of the image acquisition device to a preset multiple, and shoot the target object. The preset posture change amount is the posture change amount of the gimbal from shooting the auxiliary fixed-point marker to shooting the target object during the gimbal calibration process.

[0031] Specifically, during the calibration of the gimbal, the change in the gimbal's posture from the image acquisition device from shooting the auxiliary fixed-point marker to shooting the target object is determined. The relative positions of the auxiliary fixed-point marker and the target object in the physical space are unchanged, so the change in the gimbal's posture from shooting the auxiliary fixed-point marker to shooting the target object is also unchanged. During the actual inspection process, the initial corrected posture of the gimbal when shooting the auxiliary fixed-point marker is determined, and the gimbal is directly rotated from the initial corrected posture by a preset posture change amount, so that the target object can be within the shooting screen of the image acquisition device. Since the distance between the target object and the robot is relatively far, the zoom ratio of the image acquisition device needs to be magnified to a preset multiple. The preset multiple is the zoom ratio magnification that can be manually adjusted during the gimbal calibration process to capture a complete and clear image of the target object.

[0032] The robot shooting and positioning method provided in this embodiment corrects the posture of the gimbal by shooting auxiliary fixed-point markers multiple times, ensuring more accurate locking and shooting of the target object during high-power zoom shooting, so that the robot gimbal can locate the target object and shoot more quickly, obtain high-quality inspection data, improve inspection efficiency, and reduce labor costs.

[0033] In this embodiment, a robot shooting and positioning method is provided, which can be used in the above-mentioned computer system. Figure 2 is a flow chart of a robot shooting and positioning method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201, move the robot to a preset working position, rotate the gimbal to a preset initial posture, obtain the actual image of the auxiliary fixed-point marker, and determine the shooting deviation based on the reference image and actual image of the auxiliary fixed-point marker. The reference image is obtained by shooting the auxiliary fixed-point marker during the gimbal calibration process.

[0034] In some optional implementations, the gimbal calibration process includes: Step a1, move the robot to a preset working position, rotate the gimbal to a preset initial posture, and obtain a reference image of the auxiliary fixed-point marker.

[0035] Specifically, the gimbal calibration process is generally the calibration process of the robot and the gimbal before the robot is put into inspection. In addition, if there is a large deviation in the target object photographed during the robot inspection, the gimbal needs to be calibrated. The calibration process is a manual control process. The robot is moved to the preset working position and the gimbal is rotated to the preset initial posture. At this time, the reference image of the auxiliary fixed-point marker is obtained, and the preset working position of the robot and the preset initial posture of the gimbal are recorded.

[0036] Step a2, adjust the gimbal posture, set the zoom ratio of the image acquisition device to a preset multiple, make the target object within the shooting range of the gimbal, record the target posture of the gimbal, and determine the preset posture change amount based on the target posture and the preset initial posture.

[0037] Specifically, manually adjust the gimbal posture so that the target object can be clearly and completely captured after the zoom ratio of the image acquisition device is enlarged, and record the gimbal posture at this time as the target posture. The zoom ratio of the image acquisition device is a preset multiple, and the preset posture change amount is determined according to the target posture of the gimbal and the preset initial posture, that is, the posture change of the gimbal from shooting the auxiliary fixed-point marker to shooting the target object.

[0038] The robot shooting and positioning method provided in this embodiment calibrates the pan-tilt head at a preset working position in advance to determine the reference image and posture change, which is beneficial to compensate for the pan-tilt head posture during subsequent inspections and improves the shooting efficiency of the target object.

[0039] Specifically, the above step S201 includes: Step S2011, using the pre-trained model to determine the reference center point coordinates of the auxiliary fixed-point marker in the reference image.

[0040] Specifically, the YOLOv5 algorithm is used to find the coordinates of the reference center point of the auxiliary fixed-point marker in the reference image and record them as A (x0, y0).

[0041] In some optional implementations, the above step S2011 includes: Step b1, obtaining multiple sample images containing auxiliary fixed-point markers, and marking the boundary box of the auxiliary fixed-point marker in each sample image.

[0042] Specifically, at a low zoom ratio, multiple sample images containing auxiliary fixed-point markers are obtained to ensure that all sample images contain auxiliary fixed-point markers, and the bounding boxes of the auxiliary fixed-point markers in the sample images used for training are manually annotated. The shape of the bounding box can be a rectangle, as an example only, but not limited to this.

[0043] Step b2: training the preset model using the sample images with marked bounding boxes to obtain a pre-trained model.

[0044] Specifically, each sample image with annotated bounding boxes is divided into a training set and a test set. The training set is used to train the preset model, which is the yolov5 network framework. The parameters of the yolov5 network are updated through the back propagation algorithm to minimize the loss function. The loss function includes the bounding box coordinate error, the category prediction error and the confidence error. The test set is used to evaluate the trained yolov5 network model. According to the evaluation results, the model is optimized, the network structure is adjusted, the loss function is improved, and the training data is increased. For example, if the performance of the trained yolov5 model on the test set is not good, it may be because the amount of data in the training set is insufficient or the diversity is not enough. The diversity of the training data can be increased by adding sample data enhancement strategies (such as rotation, scaling, cropping, flipping, etc.); you can also consider using a deeper network architecture or adding more convolutional layers, or you can try to use a lighter model architecture, such as the upgrade from YOLOv5s (small model) to YOLOv5x (large model). The specific model training and optimization process is a mature existing technology and will not be repeated here.

[0045] Step b3: input the reference image into the pre-trained model to obtain the bounding box of the auxiliary fixed-point marker in the reference image, and determine the vertex coordinates of the bounding box in the reference image based on the image coordinate system.

[0046] Specifically, the reference image is input into the pre-trained model, and the pre-trained model performs target detection on the reference image. The yolov5 network model divides the reference image into multiple grids, and each grid predicts the target whose center falls within the grid. For each grid, yolov5 predicts multiple bounding boxes, and each bounding box contains the location and category probability of the target. For example, the yolov5 network model divides the reference image into S×S grids, and each grid is responsible for detecting the target whose center point falls within the grid. Yolov5 predicts multiple bounding boxes, and each network predicts the confidence score of each bounding box. The confidence score includes the possibility that the bounding box contains the target and the accuracy of the bounding box. The final target position is determined by setting the confidence threshold. For the bounding boxes of the auxiliary fixed-point markers in the detected reference image, non-maximum suppression processing is used to eliminate overlapping bounding boxes to obtain the final bounding box detection result of the auxiliary fixed-point markers.

[0047] For a rectangular bounding box, the yolov5 model can be used to predict the intersection points of each boundary, that is, the vertex coordinates of the bounding box, in the image coordinate system.

[0048] Step b4, calculating the coordinates of the reference center point according to the vertex coordinates of the bounding box in the reference image.

[0049] In some optional implementations, the above step b4 includes: Step b41, obtaining the coordinates of the upper left corner vertex and the lower right corner vertex of the bounding box in the reference image.

[0050] Specifically, the yolov5 model is used to predict the upper left vertex coordinates (xl, yl) and the lower right vertex coordinates (xr, yr) of the bounding box in the reference image.

[0051] Step b42, calculating the average value of the upper left corner vertex coordinates and the lower right corner vertex coordinates as the reference center point coordinates.

[0052] Specifically, the average value of the coordinates of the upper left vertex and the lower right vertex is calculated as: cx=(xl+xr) / 2, cy=(yl+yr) / 2, (cx, cy) is the coordinates of the reference center point.

[0053] It should be noted that the average value of the coordinates of the left side of the lower left vertex and the upper right vertex can also be calculated as the coordinates of the reference center point. The specific calculation process is not repeated here. The output of the yolov5 model is the coordinates of the reference center point of the detected auxiliary fixed-point marker, the size of the bounding box, the category confidence, etc.

[0054] The robot shooting and positioning method provided in this embodiment, after marking the bounding box of multiple sample images of auxiliary fixed-point markers, trains the preset model, and processes the reference image with the trained pre-trained model to obtain accurate vertex coordinates of the bounding box, so as to achieve the purpose of quickly determining the left side of the center point and improve the efficiency of gimbal posture correction.

[0055] Step S2012, using the pre-trained model to determine the actual center point coordinates of the auxiliary fixed-point marker in the actual image.

[0056] Specifically, the process of inputting the actual image into the pre-trained model and using the pre-trained model to determine the actual center point coordinates of the auxiliary fixed-point marker in the actual image is the same as the process of using the pre-trained model to determine the reference center point coordinates of the auxiliary fixed-point marker in the reference image in step S2011, and will not be repeated here.

[0057] Step S2013, calculating the deviation between the reference center point coordinates and the actual center point coordinates as the shooting deviation.

[0058] Specifically, in the image coordinate system, the deviation between the reference center point coordinates and the actual center point coordinates is calculated as the shooting deviation. For example, if the reference center point coordinates are (cx, cy) and the actual center point coordinates are (dx, dy), then the shooting deviation is (dx-cx, dy-cy).

[0059] The robot shooting and positioning method provided in this embodiment uses a preset model to determine the reference center point coordinates and the actual center point coordinates, and determines the shooting deviation based on this, and finds the shooting deviation in the shooting picture, so as to correct and compensate the gimbal posture accordingly, ensuring that the target object does not exceed the shooting range when shooting the target object subsequently.

[0060] Step S202, adjusting the preset initial posture of the gimbal multiple times according to the shooting deviation until there is no shooting deviation, thereby obtaining the corrected initial posture of the gimbal.

[0061] Specifically, the above step S202 includes: Step S2021, according to a preset conversion relationship between the image coordinate system and the gimbal coordinate system, the shooting deviation is converted from the image coordinate system to the gimbal coordinate system to obtain the gimbal deviation.

[0062] Specifically, the image coordinate system is a two-dimensional coordinate system used to describe the coordinates of each point in the image after the image acquisition device forms an image. The coordinate system where the point coordinates are located can be directly read from the image acquisition device. When the unit is pixel, it is called the (u, v) pixel coordinate system, and when the unit is physical size, it is called the (x, y) image coordinate system. In computer vision, the (u, v) coordinate system takes the upper left corner of the image as the origin, and the u-axis and v-axis are parallel to the two vertical edges of the image plane (the u-axis faces right and the v-axis faces downward), such as Figure 3 As shown; the (x, y) coordinate system takes the image principal point (u0, v0) as the origin, and the x-axis and y-axis are parallel to the u-axis and v-axis respectively and have the same direction. The image acquisition device can be a pan-tilt camera, which is only used as an example, but is not limited to this.

[0063] The gimbal coordinate system is a three-dimensional coordinate system established on the gimbal, which can describe the position of an object from the perspective of the gimbal camera. There is a preset conversion relationship between the image coordinate system and the gimbal coordinate system. Since the shooting deviation needs to be reduced by rotating the gimbal, the shooting deviation needs to be converted from the image coordinate system to the gimbal coordinate system to obtain the gimbal deviation in the gimbal coordinate system.

[0064] Step S2022, adjusting the preset initial posture of the gimbal multiple times according to the gimbal deviation.

[0065] Specifically, the gimbal deviation and the preset initial posture of the gimbal are both in the gimbal coordinate system, so the preset initial posture of the gimbal and the gimbal deviation are directly added to obtain a new preset initial posture. After each adjustment of the preset initial posture of the gimbal, it is necessary to re-shoot the actual image of the auxiliary fixed-point marker, calculate the shooting deviation between the actual image and the reference image again, and re-adjust the preset initial posture of the gimbal according to the shooting deviation until the shooting deviation is 0, that is, the actual center point coordinates in the actual image coincide with the reference center point coordinates in the reference image. The preset initial posture when the shooting deviation is 0 is used as the corrected initial posture of the gimbal.

[0066] The robot shooting and positioning method provided in this embodiment obtains the gimbal deviation by converting the shooting deviation from the image coordinate system to the gimbal coordinate system, which facilitates the gimbal internal control system to adjust the gimbal posture according to the gimbal deviation, thereby improving the accuracy of the gimbal posture adjustment.

[0067] In some optional implementations, in the above step S2021, the preset conversion relationship between the image coordinate system and the gimbal coordinate system is:

[0068] Where s represents the scaling factor, represents the coordinates of the target point in the image coordinate system, A represents the intrinsic parameter matrix of the gimbal camera, represents the extrinsic parameter matrix of the gimbal camera, Indicates the coordinate system of the gimbal The coordinates of the corresponding target point.

[0069] Specifically, the robot gimbal calibration process involves four coordinate systems: Pixel coordinate system (u, v), imaging plane coordinate system (x, y), gimbal coordinate system (xc, yc, zc), world coordinate system (xw, yw, zw). The transformation relationship between the world coordinate system and the gimbal coordinate system (equivalent to the robot gimbal camera coordinate system. It should be noted that during the coordinate transformation process, only the gimbal moves, and the robot does not move) is a rigid body transformation. It only requires a rotation R and a translation t for the three-dimensional point in the world coordinate system, which can be expressed as:

[0070] Among them, R and t are the external parameters of the robot gimbal camera, R represents rotation, and t represents translation.

[0071] The conversion process from the gimbal coordinate system to the imaging plane coordinate system is a conversion from three-dimensional coordinates to two-dimensional coordinates, which is also the projection perspective process. The object is projected onto the projection surface using the central projection method, thereby obtaining a single-sided projection image that is closer to the visual effect, that is, an imaging method that makes the human eye see that the scene is larger when it is near and smaller when it is far away. Figure 4 As shown, the pinhole plane (robot gimbal camera coordinate system) is between the image plane (image coordinate system) and the object point plane (chessboard plane), and the resulting image is an inverted real image.

[0072] For the convenience of mathematical description, the positions of the robot gimbal camera coordinate system and the image coordinate system are swapped to obtain Figure 5 Assume that there is a point M (X m , Y m , Z m ), the relationship between the imaging point P (xP, yP) in the image coordinate system is: , where f represents the focal length, so in the image coordinate system, the coordinates of the imaging point P are: .

[0073] like Figure 3 As shown, there is a scaling and translation between the imaging plane coordinate system (x, y) and the pixel coordinate system (u, v): , combined with the coordinates of the imaging point P converted from the gimbal coordinate system to the image coordinate system, the coordinates of the imaging point P in the pixel coordinate system are obtained as follows:

[0074] The coordinates of the imaging point P expressed in pixels are:

[0075] in, , The unit of is pixel / meter; , Indicates the focal length in the x and y directions, in pixels; Indicates the center point coordinates in pixels. The conversion relationship between the gimbal coordinate system and the pixel coordinate system is expressed as:

[0076] So the three-dimensional point in the world coordinate system and a two-dimensional point in pixel coordinates The relationship is expressed as:

[0077] Right now:

[0078] Where s represents the scaling factor, represents the coordinates of the target point in the image coordinate system, A represents the intrinsic parameter matrix of the gimbal camera, represents the extrinsic parameter matrix of the gimbal camera, Indicates the coordinate system of the gimbal The coordinates of the corresponding target point.

[0079] The robot shooting and positioning method provided in this embodiment determines the preset conversion relationship between the image coordinate system and the gimbal coordinate system through the gimbal's own parameters, which is conducive to quickly determining the gimbal deviation and improving the efficiency of gimbal posture correction and shooting the target object.

[0080] Step S203, according to the preset posture change amount, the gimbal is rotated from the calibrated initial posture, the zoom ratio of the image acquisition device is set to a preset multiple, and the target object is photographed. The preset posture change amount is the posture change amount of the gimbal from shooting the auxiliary fixed-point marker to shooting the target object during the gimbal calibration process. For details, please refer to Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0081] The robot shooting and positioning method provided in this embodiment corrects the posture of the gimbal by shooting auxiliary fixed-point markers multiple times, ensuring more accurate locking and shooting of the target object during high-power zoom shooting, so that the robot gimbal can locate the target object and shoot more quickly, obtain high-quality inspection data, improve inspection efficiency, and reduce labor costs.

[0082] In this embodiment, a robot shooting positioning device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0083] This embodiment provides a robot shooting positioning device, and the pan / tilt installed on the robot is used to support the image acquisition device, such as Figure 6 As shown, the device comprises: The shooting deviation determination module 601 is used to move the robot to a preset working position, rotate the gimbal to a preset initial posture, obtain the actual image of the auxiliary fixed-point marker, and determine the shooting deviation based on the reference image and actual image of the auxiliary fixed-point marker. The reference image is obtained by shooting the auxiliary fixed-point marker during the gimbal calibration process.

[0084] The posture correction module 602 is used to adjust the preset initial posture of the gimbal for multiple times according to the shooting deviation until there is no shooting deviation, thereby obtaining the corrected initial posture of the gimbal.

[0085] The target object shooting module 603 is used to rotate the gimbal from the calibrated initial posture according to the preset posture change amount, set the zoom ratio of the image acquisition device to a preset multiple, and shoot the target object. The preset posture change amount is the posture change amount of the gimbal from shooting the auxiliary fixed point marker to shooting the target object during the gimbal calibration process.

[0086] In some optional implementations, the shooting deviation determination module 601 includes: The reference coordinate determination unit is used to determine the reference center point coordinates of the auxiliary fixed-point marker in the reference image using a pre-trained model.

[0087] The actual coordinate determination unit is used to determine the actual center point coordinates of the auxiliary fixed-point marker in the actual image using a pre-trained model.

[0088] The deviation calculation unit is used to calculate the deviation between the reference center point coordinates and the actual center point coordinates as the shooting deviation.

[0089] In some optional implementations, the reference coordinate determination unit includes: The sample annotation subunit is used to obtain multiple sample images containing auxiliary fixed-point markers and annotate the boundary box of the auxiliary fixed-point marker in each sample image.

[0090] The model training subunit is used to train the preset model using the sample images with marked bounding boxes to obtain a pre-trained model.

[0091] The vertex coordinate determination subunit is used to input the reference image into the pre-trained model, obtain the bounding box of the auxiliary fixed-point marker in the reference image, and determine the vertex coordinates of the bounding box in the reference image based on the image coordinate system.

[0092] The reference center point calculation subunit is used to calculate the coordinates of the reference center point according to the vertex coordinates of the bounding box in the reference image.

[0093] In some optional implementations, the posture correction module 602 includes: The coordinate conversion unit is used to convert the shooting deviation from the image coordinate system to the gimbal coordinate system according to the preset conversion relationship between the image coordinate system and the gimbal coordinate system to obtain the gimbal deviation.

[0094] The posture adjustment unit is used to adjust the preset initial posture of the gimbal multiple times according to the gimbal deviation.

[0095] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0096] The robot shooting and positioning device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0097] The embodiment of the present invention also provides a computer device having the above Figure 6 The robot shooting positioning device shown.

[0098] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0099] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0100] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0101] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0102] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0103] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0104] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0105] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A robot shooting and positioning method, characterized in that: The pan-tilt platform installed on the robot is used to support the image acquisition device, and the method includes: Move the robot to a preset working position, rotate the gimbal to a preset initial position, obtain an actual image of the auxiliary fixed-point marker, and determine the shooting deviation according to the reference image and actual image of the auxiliary fixed-point marker, wherein the reference image is obtained by shooting the auxiliary fixed-point marker during the gimbal calibration process; Adjusting the preset initial posture of the gimbal multiple times according to the shooting deviation until there is no shooting deviation, thereby obtaining a corrected initial posture of the gimbal; The gimbal is rotated from the calibrated initial posture according to a preset posture change amount, the zoom ratio of the image acquisition device is set to a preset multiple, and the target object is photographed. The preset posture change amount is the posture change amount of the gimbal from shooting the auxiliary fixed-point marker to shooting the target object during the gimbal calibration process.

2. The method according to claim 1, characterized in that The gimbal calibration process includes: Move the robot to the preset working position, rotate the gimbal to the preset initial posture, and obtain the reference image of the auxiliary fixed-point marker; Adjust the gimbal posture, set the zoom ratio of the image acquisition device to a preset multiple, place the target object within the shooting range of the gimbal, record the target posture of the gimbal, and determine the preset posture change based on the target posture and the preset initial posture.

3. The method according to claim 1, characterized in that Determining the shooting deviation according to the reference image and the actual image of the auxiliary fixed-point marker includes: Determine the reference center point coordinates of the auxiliary fixed-point marker in the reference image using the pre-trained model; Determine the actual center point coordinates of the auxiliary fixed-point marker in the actual image using the pre-trained model; The deviation between the reference center point coordinates and the actual center point coordinates is calculated as the shooting deviation.

4. The method according to claim 3, characterized in that: Determining the reference center point coordinates of the auxiliary fixed-point marker in the reference image using the pre-trained model includes: Acquire multiple sample images containing auxiliary fixed-point markers, and mark the boundary box of the auxiliary fixed-point marker in each sample image; The preset model is trained using sample images with marked bounding boxes to obtain a pre-trained model; Inputting the reference image into the pre-trained model, obtaining a bounding box of the auxiliary fixed-point marker in the reference image, and determining vertex coordinates of the bounding box in the reference image based on the image coordinate system; The coordinates of the reference center point are calculated according to the coordinates of the vertices of the bounding box in the reference image.

5. The method according to claim 4, characterized in that Calculating the coordinates of the reference center point according to the vertex coordinates of the bounding box in the reference image includes: Get the coordinates of the upper left corner and the lower right corner of the bounding box in the reference image; Calculate the average value of the upper left corner vertex coordinates and the lower right corner vertex coordinates as the reference center point coordinates.

6. The method according to claim 1 or 3, characterized in that: The preset initial posture of the gimbal is adjusted multiple times according to the shooting deviation, including: According to a preset conversion relationship between the image coordinate system and the gimbal coordinate system, the shooting deviation is converted from the image coordinate system to the gimbal coordinate system to obtain the gimbal deviation; The preset initial posture of the gimbal is adjusted multiple times according to the gimbal deviation.

7. The method according to claim 6, characterized in that The preset conversion relationship between the image coordinate system and the gimbal coordinate system is: Where s represents the scaling factor, represents the coordinates of the target point in the image coordinate system, A represents the intrinsic parameter matrix of the gimbal camera, represents the extrinsic parameter matrix of the gimbal camera, Indicates the coordinate system of the gimbal The coordinates of the corresponding target point.

8. A robot shooting and positioning device, characterized in that: The pan-tilt platform installed on the robot is used to support the image acquisition device, and the device includes: A shooting deviation determination module is used to move the robot to a preset working position, rotate the gimbal to a preset initial posture, obtain an actual image of the auxiliary fixed-point marker, and determine the shooting deviation according to a reference image and an actual image of the auxiliary fixed-point marker, wherein the reference image is obtained by shooting the auxiliary fixed-point marker during the gimbal calibration process; A posture correction module, used for adjusting the preset initial posture of the gimbal for multiple times according to the shooting deviation, until there is no shooting deviation, so as to obtain a corrected initial posture of the gimbal; The target object shooting module is used to rotate the gimbal from the calibrated initial posture according to a preset posture change amount, set the zoom ratio of the image acquisition device to a preset multiple, and shoot the target object. The preset posture change amount is the posture change amount of the image acquisition device from shooting the auxiliary fixed-point marker to shooting the target object gimbal during the gimbal calibration process.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images