Control method and device of a gimbal, gimbal and storage medium

Abstract

This application provides a control method, apparatus, gimbal, and storage medium for a gimbal, belonging to the field of gimbal technology. The method includes: acquiring a visible light image including a target object, an infrared image including the target object, and laser ranging data of the gimbal relative to the target object; determining a target pose of the target object relative to the gimbal based on the visible light image, the infrared image, and the laser ranging data; determining deviation information between the current first pose of the gimbal and the target pose; and adjusting the pose of the gimbal based on the deviation information, so that the adjusted gimbal is aligned with the target object. This application significantly improves the accuracy of gimbal alignment with a target object.

Description

Technical Field

[0001] This application relates to the field of gimbal technology, and in particular to a gimbal control method, device, gimbal, and storage medium. Background Technology

[0002] With the rapid development of automation, gimbals are becoming increasingly common in daily life and industrial production. For example, inspection robots are equipped with gimbals. The inspection robot moves to a predetermined position and takes pictures of the target through the gimbal. The pictures are then uploaded to a cloud server for image analysis. Currently, gimbal alignment with the target object relies on a single vision sensor or a fixed threshold algorithm. This is susceptible to environmental interference such as sudden changes in lighting, smoke obstruction, highly reflective surfaces, and low-texture areas, which leads to gimbal attitude drift and low accuracy in aligning the gimbal with the target object.

[0003] Therefore, improving the accuracy of gimbal alignment with the target object is an urgent problem to be solved. Summary of the Invention

[0004] The main objective of this application is to provide a control method, device, gimbal, and storage medium for a gimbal, aiming to improve the accuracy of the gimbal in aligning with a target object.

[0005] Firstly, this application provides a method for controlling a gimbal, the method comprising the following steps: Acquire visible light images of the target object, infrared images of the target object, and laser ranging data of the gimbal relative to the target object; The target pose of the target object relative to the gimbal is determined based on the visible light image, the infrared image, and the laser ranging data. Determine the deviation information between the current first pose of the gimbal and the target pose; Based on the deviation information, the pose of the gimbal is adjusted so that the gimbal after the pose adjustment is aligned with the target object.

[0006] Secondly, this application also provides a control device for a gimbal, the control device for the gimbal including an acquisition module, a determination module, and an adjustment module, wherein: The acquisition module is used to acquire a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object. The determining module is used to determine the target pose of the target object relative to the gimbal based on the visible light image, the infrared image, and the laser ranging data; The determining module is further configured to determine the deviation information between the current first pose of the gimbal and the target pose; The adjustment module is used to adjust the pose of the gimbal according to the deviation information, so that the gimbal after the pose adjustment is aligned with the target object.

[0007] Thirdly, this application also provides a gimbal, the gimbal including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the gimbal control method described above.

[0008] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the gimbal control method described above.

[0009] This application provides a control method, device, gimbal, and storage medium for a gimbal. The application acquires a visible light image of a target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object. Based on the visible light image, infrared image, and laser ranging data, it determines the target pose of the target object relative to the gimbal. It then determines the deviation information between the current first pose of the gimbal and the target pose. Based on the deviation information, it adjusts the gimbal's pose so that the adjusted gimbal is aligned with the target object. This application determines the target pose of the target object relative to the gimbal using visible light images, infrared images, and laser ranging data, and accurately calculates the deviation information based on the current first pose and the target pose. Then, based on the deviation information, it adjusts the gimbal's pose so that the adjusted gimbal is aligned with the target object, greatly improving the accuracy of gimbal alignment. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a gimbal control method provided in an embodiment of this application; Figure 2 for Figure 1 A flowchart illustrating the sub-steps of the gimbal control method in the diagram; Figure 3 for Figure 1 A flowchart illustrating the sub-steps of the gimbal control method in the diagram; Figure 4 for Figure 3 A flowchart illustrating the sub-steps of the gimbal control method in the diagram; Figure 5 A schematic block diagram of a gimbal control device provided in an embodiment of this application; Figure 6 This is a schematic block diagram of a gimbal structure provided for an embodiment of this application.

[0012] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0015] With the rapid development of automation, gimbals are becoming increasingly common in daily life and industrial production. For example, inspection robots are equipped with gimbals. The inspection robot moves to a predetermined position and takes pictures of the target through the gimbal. The pictures are then uploaded to a cloud server for image analysis. Currently, gimbal alignment with the target object relies on a single vision sensor or a fixed threshold algorithm. This is susceptible to environmental interference such as sudden changes in lighting, smoke obstruction, highly reflective surfaces, and low-texture areas, which leads to gimbal attitude drift and low accuracy in aligning the gimbal with the target object.

[0016] To address the aforementioned problems, embodiments of this application provide a gimbal control method, apparatus, gimbal, and storage medium. The gimbal control method includes acquiring a visible light image including a target object, an infrared image including the target object, and laser ranging data of the gimbal relative to the target object; determining a target pose of the target object relative to the gimbal based on the visible light image, the infrared image, and the laser ranging data; determining deviation information between the current first pose of the gimbal and the target pose; and adjusting the pose of the gimbal based on the deviation information, so that the adjusted gimbal is aligned with the target object.

[0017] The control method of this gimbal can be applied to gimbals, including ball gimbals, 3D gimbals, hydraulic gimbals, and cantilever gimbals, etc. The control method of this gimbal can also be applied to equipment such as inspection robots.

[0018] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0019] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a gimbal control method provided in an embodiment of this application.

[0020] like Figure 1 As shown, the control method of the gimbal includes steps S101 to S104.

[0021] Step S101: Acquire a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object.

[0022] The control of the gimbal can be applied to the gimbal itself, which is equipped with a visible light camera, an infrared camera, and a laser rangefinder. The visible light camera is used to acquire visible light images of the target object, the infrared camera is used to acquire infrared images of the target object, and the laser rangefinder is used to acquire the distance between the gimbal and the target object. The visible light camera, infrared camera, and laser rangefinder can be configured according to actual conditions, and this embodiment does not impose specific limitations on them.

[0023] It should be noted that the target object can be determined according to the actual situation, and this application does not make specific limitations on it. For example, the target object can be a workpiece, etc.

[0024] In some embodiments, a visible light image is obtained by a visible light camera capturing an image of the target object, an infrared image is obtained by an infrared camera capturing an infrared image of the target object, and a laser rangefinder is used to measure the distance to the target object to obtain laser ranging data of the gimbal relative to the target object.

[0025] For example, when alignment is required, the gimbal controls the visible light camera configured on the gimbal to acquire a visible light image of the target object, controls the infrared camera to acquire an infrared image of the target object, and controls the laser range sensor to measure the distance to the target object, thereby obtaining laser range data of the gimbal relative to the target object.

[0026] For example, this is applied to an inspection robot equipped with a gimbal. The gimbal controls the robot to move to the inspection position. During the secondary alignment and detection by the gimbal, the visible light camera configured on the gimbal is controlled to acquire an image of the target object, the infrared camera is controlled to acquire an infrared image of the target object, and the laser range sensor is controlled to measure the distance to the target object, thereby obtaining laser range data of the gimbal relative to the target object.

[0027] Step S102: Determine the target pose of the target object relative to the gimbal based on the visible light image, the infrared image, and the laser ranging data.

[0028] The target pose refers to the target position and orientation of the target object relative to the gimbal.

[0029] In some embodiments, such as Figure 2 As shown, step S102 includes sub-steps S1021 to S1023.

[0030] Sub-step S1021: Spatiotemporally align the visible light image, the infrared image, and the laser ranging data to obtain the target visible light image, the target infrared image, and the target ranging data.

[0031] A pre-trained, convergent neural network model is obtained using a preset feature matching model. Dense feature maps are extracted from visible light and infrared images using this model, resulting in visible light dense feature images and infrared dense feature images. Feature matching is then performed on these images to calculate the homography transformation matrix between them, thereby registering the visible light and infrared images to a unified coordinate system. The laser ranging data is then projected onto this coordinate system using a preset extrinsic parameter matrix to obtain the target visible light image, target infrared image, and target ranging data. This preset feature matching model can be a multimodal large model, such as Qwen3-VL. This model is an existing end-to-end large model, meaning its training process uses conventional techniques, which will not be elaborated upon in this application.

[0032] Sub-step S1022: Extract and stitch features from the target visible light image, the target infrared image, and the target ranging data to obtain the target feature vector.

[0033] In some embodiments, geometric features are extracted from the target visible light image to obtain a geometric feature vector; temperature features are extracted from the target infrared image to obtain a temperature feature vector; depth features are extracted from the target ranging data to obtain a depth feature vector; and the geometric feature vector, temperature feature vector, and depth feature vector are concatenated to obtain the target feature vector. By performing feature extraction and feature concatenation on the target visible light image, target infrared image, and target ranging data, the target feature vector can be accurately obtained.

[0034] In some embodiments, geometric feature extraction of the target visible light image to obtain a geometric feature vector can be performed by extracting the target contour, edge, and corner features from the target visible light image to obtain a geometric feature vector.

[0035] In some embodiments, the method for extracting temperature features from the target infrared image to obtain a temperature feature vector can be as follows: extracting high-temperature region, temperature gradient, and target region mask features from the target infrared image to obtain a temperature feature vector.

[0036] In some embodiments, the method for extracting depth features from target ranging data to obtain a depth feature vector can be as follows: extracting depth features of target, distance abrupt change points, and occlusion boundaries from target ranging data to obtain a depth feature vector.

[0037] In some embodiments, the geometric feature vector, temperature feature vector, and depth feature vector can be concatenated to obtain the target feature vector by: concatenating the geometric feature vector, temperature feature vector, and depth feature vector along the feature dimension to form a multimodal feature vector, thereby obtaining the target feature vector, wherein the number of channels of the target feature vector is the sum of the three.

[0038] Sub-step S1023: Perform pose calculation based on the target feature vector using a preset pose calculation model to obtain the target pose of the target object relative to the gimbal.

[0039] The preset pose calculation model is a pre-trained convergent neural network model. The preset pose calculation model can be selected according to the actual situation. This application does not make specific limitations on it. For example, the preset pose calculation model can be a Transformer model. The preset pose calculation model is obtained by supervising the training of the Transformer model based on the sample dataset.

[0040] In some embodiments, the target feature vector is output with the three-dimensional pose parameters of the target object by a preset pose calculation model to obtain the target pose of the target object relative to the gimbal. The target pose includes the target center coordinates and the target pose angle. The target center coordinates are the center coordinates of the target object in the image. The target pose angle is the angular deviation between the gimbal optical axis and the line connecting the center of the gimbal and the center of the target object, with the gimbal coordinates as a reference.

[0041] Step S103: Determine the deviation information between the current first pose of the gimbal and the target pose.

[0042] The target pose includes target center coordinates and target attitude angles. The target center coordinates are the center coordinates of the target object in the image. The target attitude angle is the angular deviation between the gimbal optical axis and the line connecting the center of the gimbal and the center of the target object, with reference to the gimbal coordinates. The target attitude angle includes the target azimuth angle and the target pitch angle. The first pose includes first center coordinates and first attitude angles. The first center coordinates are the current center coordinates of the target object in the image. The first attitude angle is the current angular deviation between the gimbal optical axis and the line connecting the center of the gimbal and the center of the target object, with reference to the gimbal coordinates. The first attitude angle includes the first azimuth angle and the first pitch angle.

[0043] In some embodiments, the displacement deviation between the target center coordinates and the first center coordinates is calculated to obtain the horizontal offset and the vertical offset; the angle difference between the target attitude angle and the first attitude angle is calculated to obtain the angle deviation value.

[0044] In some embodiments, the displacement deviation between the target center coordinates and the first center coordinates is calculated to obtain the horizontal and vertical offsets. The target center coordinates include the target center horizontal coordinate and the target center vertical coordinate, and the first center coordinates include the first center horizontal coordinate and the first center vertical coordinate. The horizontal offset is obtained by subtracting the first center horizontal coordinate from the target center horizontal coordinate and then multiplying it by a preset pixel distance coefficient; the vertical offset is obtained by subtracting the first center vertical coordinate from the target center vertical coordinate and then multiplying it by the preset pixel distance coefficient. The preset pixel distance coefficient is set according to actual conditions, and this embodiment does not specifically limit it. For example, the preset pixel distance coefficient is a conversion coefficient between pixel size and actual distance. For example, the pixel size is the ratio of the camera pixel size to the lens focal length, and the actual distance is the distance collected by the laser rangefinder. For example, if the camera pixel size is 4.8μm, the lens focal length is 8mm, and the actual distance is 1.5m, then 4.8*8 / 1.5=0.9.

[0045] In some embodiments, the angle difference between the target attitude angle and the first attitude angle can be calculated to obtain the angle deviation value. The target attitude angle includes the target azimuth angle and the target pitch angle; the first attitude angle includes the first azimuth angle and the first pitch angle; the target azimuth angle is subtracted from the first azimuth angle to obtain the azimuth angle deviation value, and the target pitch angle is subtracted from the first pitch angle to obtain the pitch angle deviation value.

[0046] In some embodiments, the current distance value between the gimbal and the target object is obtained, and the target distance value between the gimbal and the target object is obtained; the target distance value is subtracted from the current distance value to obtain the distance deviation value. The target distance value is obtained by querying a preset mapping table between the first pose and the target pose. That is, the target distance value is retrieved from the preset mapping table based on the first pose and the target pose. This mapping table is established based on the first pose, the target pose, and the target distance value. The mapping table can be established according to actual conditions, and this application does not specifically limit its creation.

[0047] Step S104: Adjust the pose of the gimbal according to the deviation information so that the gimbal after the pose adjustment is aligned with the target object.

[0048] In some embodiments, such as Figure 3 As shown, step S104 includes sub-steps S1041 to S1042.

[0049] Sub-step S1041: Determine the target deviation value based on the distance deviation value, the vertical offset, the horizontal offset, and the angle deviation value.

[0050] In some embodiments, the angle deviation value is multiplied by a preset conversion coefficient to obtain the candidate angle deviation value; the distance deviation value, vertical offset, horizontal offset and candidate angle deviation value are squared respectively to obtain the target distance deviation value, target vertical offset, target horizontal offset and target angle deviation value; the target distance deviation value, target vertical offset, target horizontal offset and target angle deviation value are summed and then the square root is taken to obtain the target deviation value.

[0051] In some embodiments, the method for obtaining candidate angle deviation values ​​by multiplying the angle deviation value by a preset conversion coefficient can be as follows: the angle deviation value includes an azimuth deviation value and a pitch deviation value; the azimuth deviation value is multiplied by the preset conversion coefficient to obtain a first candidate angle deviation value; the pitch deviation value is multiplied by the preset conversion coefficient to obtain a second candidate angle deviation value. The preset conversion coefficient is set according to actual conditions, and this application embodiment does not specifically limit it.

[0052] In some embodiments, the method for squaring the candidate angle deviation value can be as follows: squaring the first candidate angle deviation value and the second candidate angle deviation value respectively to obtain the first target angle deviation value and the second target angle deviation value, and then adding the first target angle deviation value and the second target angle deviation value to obtain the target angle deviation value.

[0053] Sub-step S1042: In response to the target deviation value being greater than or equal to a preset deviation value, adjust the pose of the gimbal according to the vertical offset, the horizontal offset, and the angle deviation value.

[0054] The preset deviation value can be set according to the actual situation. This application embodiment does not make a specific limitation on this. For example, the preset deviation value can be set to 0.5mm.

[0055] In some embodiments, in response to a target deviation value being greater than or equal to a preset deviation value, a motor control command is generated based on the vertical offset, horizontal offset, and angular deviation value, and the motor of the gimbal is controlled to perform the corresponding action according to the motor control command, so as to adjust the attitude of the gimbal.

[0056] In some embodiments, generating motor control commands based on vertical offset, horizontal offset, and angular deviation values ​​can be achieved by: obtaining a preset mapping table of vertical offset, horizontal offset, and angular deviation values ​​with motor control commands; and querying the motor control command matching the vertical offset, horizontal offset, and angular deviation values ​​from this mapping table. This mapping table is pre-established based on the preset vertical offset, horizontal offset, angular deviation values, and motor control commands, and this application does not impose specific limitations on it. The mapping table allows for accurate retrieval of motor control commands.

[0057] It should be noted that the motor control commands are generated based on the linear mapping relationship (LUT lookup table) between the vertical offset, horizontal offset and angular deviation values. The LUT is calibrated at the factory and supports online updates.

[0058] In some embodiments, in response to a target deviation value being less than a preset deviation value, it is determined that the gimbal and the target object are aligned.

[0059] In some embodiments, such as Figure 4 As shown, sub-step S1042 includes sub-steps S1042a to S1042c.

[0060] Sub-step S1042a: Control the laser rangefinder in the gimbal to measure the distance and obtain the second distance value.

[0061] After the gimbal completes the adjustment according to the motor control command, the laser rangefinder in the gimbal measures the distance to obtain the second distance value.

[0062] Sub-step S1042b: In response to the difference between the first distance value and the second distance value in the laser ranging data being greater than a preset difference, the attitude of the gimbal is adjusted back to the first pose, and the process returns to the step of acquiring a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object.

[0063] The first distance value is the distance between the gimbal and the target object before gimbal adjustment. This preset difference can be set according to the actual situation, and this application embodiment does not specifically limit it. For example, the preset difference can be set to 1mm.

[0064] In some embodiments, in response to the difference between a first distance value and a second distance value in the laser ranging data exceeding a preset difference, the gimbal's attitude is adjusted back to the first pose, and the process returns to acquiring a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object, to readjust the gimbal's pose. If the gimbal adjustment is significant, the current adjustment is abandoned, and data is reacquired before further adjustment to improve the accuracy of the gimbal adjustment.

[0065] Sub-step S1042c: In response to the difference between the first distance value and the second distance value in the laser ranging data being less than or equal to a preset difference, the pose of the gimbal is further adjusted according to the difference between the target distance value and the second distance value.

[0066] The target distance value is the final distance between the gimbal and the target object.

[0067] In some embodiments, in response to the difference between a first distance value and a second distance value in the laser ranging data being less than or equal to a preset difference, the difference between the target distance value and the second distance value is calculated to obtain the distance value to be compensated, and a preset gimbal adjustment model is obtained. This gimbal adjustment model is a pre-trained convergent neural network model, which is obtained by pre-training the neural network model based on multiple sample data, including the distance value to be compensated, the adjustment step size, and labeled gimbal adjustment instructions. Each training iteration trains the convolutional neural network model based on the distance value to be compensated, the adjustment step size, and the labeled gimbal adjustment instructions until a convergent gimbal adjustment model is obtained. It should be noted that training the convolutional neural network model is a conventional technique, and this application does not specifically limit its use. The preset gimbal adjustment model is used to generate gimbal control instructions based on the distance value to be compensated and the adjustment step size, resulting in a target gimbal control instruction. The gimbal is then adjusted according to this target gimbal control instruction.

[0068] The system continues to reacquire visible light images, infrared images, and laser ranging data of the target object relative to the gimbal. Based on these data, the target pose of the target object relative to the gimbal is determined. The deviation information between the current first pose of the gimbal and the target pose is determined, including distance deviation, vertical offset, horizontal offset, and angular deviation. Based on these deviations, a target deviation value is determined. If the target deviation value is less than a preset deviation value, the gimbal and target object are re-evaluated. Alignment is complete. In response to a target deviation value being greater than or equal to a preset deviation value, the laser rangefinder in the gimbal measures the distance to obtain a third distance value. The difference between the target distance value and the third distance value is calculated to obtain the distance to be compensated. A gimbal control command is generated using this preset gimbal adjustment model based on the distance to be compensated and the adjustment step size, resulting in a target gimbal control command. The adjustment step size in this adjustment process can be smaller than the previous adjustment step size to improve the accuracy of the gimbal adjustment. The gimbal is adjusted according to this target gimbal control command, and the aforementioned steps are repeated until the target deviation value is less than the preset deviation value. By repeatedly fine-tuning the gimbal, accurate alignment between the gimbal and the target object can be achieved, greatly improving the accuracy of gimbal alignment.

[0069] It should be noted that the iteration stops when the aforementioned alignment steps have been repeated a preset number of times, thus avoiding infinitely looping fine-tuning.

[0070] The gimbal control method provided in the above embodiments acquires a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object; determines the target pose of the target object relative to the gimbal based on the visible light image, infrared image, and laser ranging data; determines the deviation information between the current first pose of the gimbal and the target pose; and adjusts the pose of the gimbal based on the deviation information so that the adjusted gimbal is aligned with the target object. In this application, the target pose of the target object relative to the gimbal is determined using visible light images, infrared images, and laser ranging data, and the deviation information can be accurately calculated based on the current first pose and the target pose of the gimbal; then, the pose of the gimbal is adjusted based on the deviation information so that the adjusted gimbal is aligned with the target object, greatly improving the accuracy of gimbal alignment with the target object.

[0071] Please see Figure 5 , Figure 5 This is a schematic block diagram of a gimbal control device provided in an embodiment of this application.

[0072] like Figure 5As shown, the control device 200 of the gimbal includes an acquisition module 210, a determination module 220, and an adjustment module 230, wherein: The acquisition module 210 is used to acquire a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object. The determining module 220 is used to determine the target pose of the target object relative to the gimbal based on the visible light image, the infrared image and the laser ranging data; The determining module 220 is further configured to determine the deviation information between the current first pose of the gimbal and the target pose; The adjustment module 230 is used to adjust the pose of the gimbal according to the deviation information, so that the gimbal after the pose adjustment is aligned with the target object.

[0073] In some embodiments, the determining module 220 is further configured to: Spatiotemporal alignment of the visible light image, the infrared image, and the laser ranging data is performed to obtain the target visible light image, the target infrared image, and the target ranging data. Feature extraction and feature concatenation are performed on the target visible light image, the target infrared image, and the target ranging data to obtain the target feature vector; The target pose of the target object relative to the gimbal is obtained by performing pose calculation based on the target feature vector using a preset pose calculation model.

[0074] In some embodiments, the determining module 220 is further configured to: Geometric features are extracted from the target visible light image to obtain a geometric feature vector; Temperature features are extracted from the infrared image of the target to obtain a temperature feature vector; Depth features are extracted from the target ranging data to obtain a depth feature vector; The geometric feature vector, the temperature feature vector, and the depth feature vector are concatenated to obtain the target feature vector.

[0075] In some embodiments, the determining module 220 is further configured to: The displacement deviation between the target center coordinates and the first center coordinates is calculated to obtain the horizontal offset and the vertical offset; The angle difference between the target attitude angle and the first attitude angle is calculated to obtain the angle deviation value.

[0076] In some embodiments, the adjustment module 230 is further configured to: The target deviation value is determined based on the distance deviation value, the vertical offset, the horizontal offset, and the angle deviation value. In response to the target deviation value being greater than or equal to a preset deviation value, the pose of the gimbal is adjusted according to the vertical offset, the horizontal offset, and the angular deviation value.

[0077] In some embodiments, the adjustment module 230 is further configured to: Multiply the angle deviation value by a preset conversion coefficient to obtain the candidate angle deviation value; The target distance deviation value, target vertical offset, target horizontal offset, and target angle deviation value are calculated by squaring the distance deviation value, the vertical offset value, the horizontal offset value, and the angle deviation value, respectively. The target deviation value is obtained by summing the target distance deviation value, the target vertical offset, the target horizontal offset, and the target angle deviation value, and then taking the square root.

[0078] In some embodiments, the adjustment module 230 is further configured to: The laser rangefinder in the gimbal is controlled to measure the distance and obtain a second distance value. In response to the fact that the difference between the first distance value and the second distance value in the laser ranging data is greater than a preset difference, the attitude of the gimbal is adjusted back to the first pose, and the process returns to the steps of acquiring a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object. In response to the difference between the first distance value and the second distance value in the laser ranging data being less than or equal to a preset difference, the pose of the gimbal is further adjusted based on the difference between the target distance value and the second distance value.

[0079] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the above-mentioned gimbal control device can be referred to the corresponding process in the aforementioned gimbal control method embodiment, and will not be repeated here.

[0080] Please see Figure 6 , Figure 6 This is a schematic block diagram illustrating the structure of a gimbal, provided as an embodiment of this application. like Figure 6 As shown, the gimbal 300 includes a processor 302 and a memory 303 connected via a system bus 301, wherein the memory 303 may include a storage medium and internal memory.

[0081] The storage medium can store a computer program. This computer program includes program instructions that, when executed, cause the processor to perform any control method for the gimbal.

[0082] Processor 302 provides computing and control capabilities to support the operation of the entire gimbal.

[0083] The internal memory provides an environment for the execution of computer programs stored in the storage medium. When the computer program is executed by the processor, it enables the processor to execute any control method of the gimbal.

[0084] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the gimbal to which the solution of this application is applied. A specific gimbal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0085] It should be understood that processor 302 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.

[0086] In one embodiment, the processor 302 is configured to run a computer program stored in a memory to perform the following steps: Acquire visible light images of the target object, infrared images of the target object, and laser ranging data of the gimbal relative to the target object; The target pose of the target object relative to the gimbal is determined based on the visible light image, the infrared image, and the laser ranging data. Determine the deviation information between the current first pose of the gimbal and the target pose; Based on the deviation information, the pose of the gimbal is adjusted so that the gimbal after the pose adjustment is aligned with the target object.

[0087] In one embodiment, when the processor 302 determines the target pose of the target object relative to the gimbal based on the visible light image, the infrared image, and the laser ranging data, it is configured to: Spatiotemporal alignment of the visible light image, the infrared image, and the laser ranging data is performed to obtain the target visible light image, the target infrared image, and the target ranging data. Feature extraction and feature concatenation are performed on the target visible light image, the target infrared image, and the target ranging data to obtain the target feature vector; The target pose of the target object relative to the gimbal is obtained by performing pose calculation based on the target feature vector using a preset pose calculation model.

[0088] In one embodiment, when the processor 302 performs feature extraction and feature concatenation on the target visible light image, the target infrared image, and the target ranging data to obtain a multimodal feature vector, it is used to: Geometric features are extracted from the target visible light image to obtain a geometric feature vector; Temperature features are extracted from the infrared image of the target to obtain a temperature feature vector; Depth features are extracted from the target ranging data to obtain a depth feature vector; The geometric feature vector, the temperature feature vector, and the depth feature vector are concatenated to obtain the target feature vector.

[0089] In one embodiment, the target pose includes target center coordinates and target attitude angles, and the first pose includes first center coordinates and first attitude angles; when the processor 302 implements the process of determining the deviation information between the current first pose of the gimbal and the target pose, it is used to: The displacement deviation between the target center coordinates and the first center coordinates is calculated to obtain the horizontal offset and the vertical offset; The angle difference between the target attitude angle and the first attitude angle is calculated to obtain the angle deviation value.

[0090] In one embodiment, when the processor 302 implements the deviation information, including distance deviation value, vertical offset, horizontal offset, and angle deviation value, and adjusts the pose of the gimbal based on the deviation information, it is configured to: The target deviation value is determined based on the distance deviation value, the vertical offset, the horizontal offset, and the angle deviation value. In response to the target deviation value being greater than or equal to a preset deviation value, the pose of the gimbal is adjusted according to the vertical offset, the horizontal offset, and the angular deviation value.

[0091] In one embodiment, when the processor 302 determines the target deviation value based on the distance deviation value, the vertical offset, the horizontal offset, and the angle deviation value, it is configured to: Multiply the angle deviation value by a preset conversion coefficient to obtain the candidate angle deviation value; The target distance deviation value, target vertical offset, target horizontal offset, and target angle deviation value are calculated by squaring the distance deviation value, the vertical offset value, the horizontal offset value, and the angle deviation value, respectively. The target deviation value is obtained by summing the target distance deviation value, the target vertical offset, the target horizontal offset, and the target angle deviation value, and then taking the square root.

[0092] In one embodiment, after adjusting the pose of the gimbal based on the vertical offset, the horizontal offset, and the angle deviation value, the processor 302 is further configured to: The laser rangefinder in the gimbal is controlled to measure the distance and obtain a second distance value. In response to the fact that the difference between the first distance value and the second distance value in the laser ranging data is greater than a preset difference, the attitude of the gimbal is adjusted back to the first pose, and the process returns to the steps of acquiring a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object. In response to the difference between the first distance value and the second distance value in the laser ranging data being less than or equal to a preset difference, the pose of the gimbal is further adjusted based on the difference between the target distance value and the second distance value.

[0093] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the gimbal described above can be referred to the corresponding process in the aforementioned gimbal control method embodiments, and will not be repeated here.

[0094] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can refer to various embodiments of the gimbal control method of this application.

[0095] The computer-readable storage medium can be the internal storage unit of the gimbal described in the foregoing embodiments, such as the hard drive or memory of the gimbal. The computer-readable storage medium can be non-volatile or volatile. Alternatively, the computer-readable storage medium can be an external storage device of the gimbal, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the gimbal.

[0096] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to include the plural forms.

[0097] It should also be understood that the term "and / or" as used in this specification refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0098] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for controlling a gimbal, characterized in that, include: Acquire visible light images of the target object, infrared images of the target object, and laser ranging data of the gimbal relative to the target object; The target pose of the target object relative to the gimbal is determined based on the visible light image, the infrared image, and the laser ranging data. Determine the deviation information between the current first pose of the gimbal and the target pose; Based on the deviation information, the pose of the gimbal is adjusted so that the gimbal after the pose adjustment is aligned with the target object.

2. The gimbal control method as described in claim 1, characterized in that, Determining the target pose of the target object relative to the gimbal based on the visible light image, the infrared image, and the laser ranging data includes: Spatiotemporal alignment of the visible light image, the infrared image, and the laser ranging data is performed to obtain the target visible light image, the target infrared image, and the target ranging data. Feature extraction and feature concatenation are performed on the target visible light image, the target infrared image, and the target ranging data to obtain the target feature vector; The target pose of the target object relative to the gimbal is obtained by performing pose calculation based on the target feature vector using a preset pose calculation model.

3. The gimbal control method as described in claim 2, characterized in that, The step of extracting and concatenating features from the visible light image, the infrared image, and the ranging data of the target to obtain a multimodal feature vector includes: Geometric features are extracted from the target visible light image to obtain a geometric feature vector; Temperature features are extracted from the infrared image of the target to obtain a temperature feature vector; Depth features are extracted from the target ranging data to obtain a depth feature vector; The geometric feature vector, the temperature feature vector, and the depth feature vector are concatenated to obtain the target feature vector.

4. The gimbal control method as described in claim 1, characterized in that, The target pose includes target center coordinates and target attitude angles, and the first pose includes first center coordinates and first attitude angles; determining the deviation information between the current first pose of the gimbal and the target pose includes: The displacement deviation between the target center coordinates and the first center coordinates is calculated to obtain the horizontal offset and the vertical offset; The angle difference between the target attitude angle and the first attitude angle is calculated to obtain the angle deviation value.

5. The control method for a gimbal as described in claim 1, characterized in that, The deviation information includes distance deviation, vertical offset, horizontal offset, and angle deviation. Adjusting the gimbal's pose based on the deviation information includes: The target deviation value is determined based on the distance deviation value, the vertical offset, the horizontal offset, and the angle deviation value. In response to the target deviation value being greater than or equal to a preset deviation value, the pose of the gimbal is adjusted according to the vertical offset, the horizontal offset, and the angular deviation value.

6. The control method for a gimbal as described in claim 5, characterized in that, Determining the target deviation value based on the distance deviation value, the vertical offset, the horizontal offset, and the angle deviation value includes: Multiply the angle deviation value by a preset conversion coefficient to obtain the candidate angle deviation value; The target distance deviation value, target vertical offset, target horizontal offset, and target angle deviation value are calculated by squaring the distance deviation value, the vertical offset value, the horizontal offset value, and the angle deviation value, respectively. The target deviation value is obtained by summing the target distance deviation value, the target vertical offset, the target horizontal offset, and the target angle deviation value, and then taking the square root.

7. The gimbal control method as described in claim 5, characterized in that, After adjusting the pose of the gimbal based on the vertical offset, the horizontal offset, and the angular deviation value, the method further includes: The laser rangefinder in the gimbal is controlled to measure the distance and obtain a second distance value. In response to the fact that the difference between the first distance value and the second distance value in the laser ranging data is greater than a preset difference, the attitude of the gimbal is adjusted back to the first pose, and the process returns to the steps of acquiring a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object. In response to the difference between the first distance value and the second distance value in the laser ranging data being less than or equal to a preset difference, the pose of the gimbal is further adjusted based on the difference between the target distance value and the second distance value.

8. A control device for a gimbal, characterized in that, The control device for the gimbal includes an acquisition module, a determination module, and an adjustment module, wherein: The acquisition module is used to acquire a visible light image of the target object, an infrared image of the target object, and laser ranging data of the gimbal relative to the target object. The determining module is used to determine the target pose of the target object relative to the gimbal based on the visible light image, the infrared image, and the laser ranging data; The determining module is further configured to determine the deviation information between the current first pose of the gimbal and the target pose; The adjustment module is used to adjust the pose of the gimbal according to the deviation information, so that the gimbal after the pose adjustment is aligned with the target object.

9. A gimbal, characterized in that, The gimbal includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the gimbal control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the gimbal control method as described in any one of claims 1 to 7.

Images