Inspection control method and device and electronic equipment

Abstract

The invention discloses an inspection control method and device and electronic equipment. The method comprises the following steps: carrying out multiple times of image acquisition and image analysis on the same relative motion target based on a pan-tilt camera carried on an inspection vehicle to obtain a plurality of detection results; carrying out streamer analysis on the plurality of detection results by adopting an optical flow tracking method to obtain the movement speed of the target and the position of the target in the visual field, and further determining the total rotation angle of the inspection vehicle and the pan-tilt camera when the target is ensured to be in the center of the visual field; based on the total rotation angle and the speed ratio of the total rotation angle and the speed ratio, rotation angles corresponding to the inspection vehicle and the pan-tilt camera are determined; and based on the respective rotation angles of the inspection vehicle and the pan-tilt camera, controlling the inspection vehicle and the pan-tilt camera to rotate simultaneously to ensure that the target is in the center of the view. According to the invention, the technical problems of target loss and poor image reliability caused by too fast relative movement speed between the inspection vehicle and the target in the inspection process are solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent inspection, and in particular to an inspection control method, device and electronic equipment. Background Art

[0002] In the field of equipment inspection, especially in power, industry and infrastructure maintenance, pan-tilt cameras, as key visual perception tools, face multiple technical challenges. For example, in the process of target tracking based on a robot pan-tilt, the adjustment of rotation speed is a key issue. There is no accurate estimation of the speed of the tracked object, resulting in a faster or slower tracking speed of the pan-tilt. When the target moves too fast, the pan-tilt may not be able to rotate in time and thus fail to lock the target; when the target moves slowly, the pan-tilt may rotate too fast, causing the target to quickly move out of the field of view. Once the target leaves the field of view, it may be difficult for the system to find the target again. Sometimes when the pan-tilt rotates too fast to track the target, motion blur may occur, resulting in a decrease in image quality.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a patrol control method, device and electronic equipment to at least solve the technical problems of target loss and poor image reliability caused by the excessively fast relative movement speed between the patrol vehicle and the target during the patrol process.

[0005] According to one aspect of an embodiment of the present invention, a patrol control method is provided, comprising: based on a pan-tilt camera carried on a patrol vehicle, performing multiple image acquisition and image analysis on the same relatively moving target to obtain multiple detection results, wherein the relatively moving target is used to indicate a target that is in relative motion with the patrol vehicle; performing optical flow analysis on the multiple detection results using an optical flow tracking method to obtain a movement speed and a position in a field of view of the same relatively moving target, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; based on the movement speed and the position in the field of view, determining a total rotation angle of the patrol vehicle and the pan-tilt, wherein the total rotation angle is the total angle that the patrol vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center position of the field of view; based on the total rotation angle and the speed ratio of the two, determining the rotation angles corresponding to the patrol vehicle and the pan-tilt camera respectively, wherein the speed ratio of the two is the speed ratio between the patrol vehicle and the pan-tilt camera; based on the rotation angles corresponding to the patrol vehicle and the pan-tilt camera respectively, controlling the rotation of the patrol vehicle and the pan-tilt camera.

[0006] According to another aspect of an embodiment of the present invention, there is also provided an inspection control device, comprising: an acquisition module, for performing multiple image acquisition and image analysis on the same relatively moving target based on a pan-tilt camera carried on an inspection vehicle, to obtain multiple detection results, wherein the relatively moving target is used to indicate a target that is in relative motion with the inspection vehicle; a light flow analysis module, for performing light flow analysis on the multiple detection results using an optical flow tracking method, to obtain a movement speed and a position in a field of view of the same relatively moving target, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; a first angle determination module, for determining a first angle based on the movement speed; and the position in the field of view, determining the total rotation angle of the inspection vehicle and the pan-tilt camera, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate when keeping the same relative moving target at the center position of the field of view; a second angle determination module, used to determine the corresponding rotation angles of the inspection vehicle and the pan-tilt camera respectively based on the total rotation angle and the speed ratio of the two, wherein the speed ratio of the two is the speed ratio between the inspection vehicle and the pan-tilt camera; a control module, used to control the rotation of the inspection vehicle and the pan-tilt camera based on the corresponding rotation angles of the inspection vehicle and the pan-tilt camera respectively.

[0007] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the inspection control methods described.

[0008] In an embodiment of the present invention, based on a pan-tilt camera carried on an inspection vehicle, multiple image acquisitions and image analyses are performed on the same relatively moving target to obtain multiple detection results, wherein the relatively moving target is used to indicate a target that is in relative motion with the inspection vehicle; an optical flow tracking method is used to perform optical flow analysis on multiple detection results to obtain a movement speed and a position in a field of view of the same relatively moving target, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; based on the movement speed and the position in the field of view, the total rotation angle of the inspection vehicle and the pan-tilt is determined, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center position of the field of view; based on the total rotation The inspection vehicle and the pan-tilt camera are controlled based on the rotation angles corresponding to the inspection vehicle and the pan-tilt camera, and the optical flow tracking method is used to track and locate the target in the image, and the angle of the inspection vehicle and the pan-tilt camera are accurately adjusted synchronously based on the determined movement speed, thereby achieving the technical effect of improving the adjustment accuracy of the robot pan-tilt, and then improving the image acquisition quality and image reliability of the robot pan-tilt, and avoiding target loss, thereby solving the technical problem of target loss and poor image reliability caused by the excessive relative movement speed between the inspection vehicle and the target during the inspection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0010] Figure 1 is a flow chart of a patrol control method according to an embodiment of the present invention;

[0011] Figure 2 is a schematic diagram of a patrol control device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0013] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0014] First, to facilitate understanding of the embodiments of the present invention, some terms or nouns involved in the present invention are explained below:

[0015] Gamma adjustment algorithm, a technology used in image processing, is mainly used to correct the brightness and contrast of images to improve the image display effect under different lighting conditions. The principle of gamma correction is based on the nonlinear characteristics of human eye's perception of brightness, that is, the human eye is more sensitive to brightness changes in darker environments, and the perception of brightness changes is weakened in brighter environments. This characteristic is particularly important in image brightness adjustment, because simple linear brightness adjustment may not accurately reflect the true perception of the human eye under different lighting conditions.

[0016] In the field of equipment inspection, especially in power, industry and infrastructure maintenance, pan-tilt cameras, as key visual perception tools, face multiple technical challenges. On the one hand, in the process of target tracking based on a robot pan-tilt, the adjustment of the rotation speed is a key issue. There is no accurate estimation of the speed of the tracking object (such as the equipment to be tested, staff, etc.), resulting in a faster or slower tracking speed of the pan-tilt. When the target moves too fast, the pan-tilt may not be able to rotate in time, and thus cannot lock the target; when the target moves slowly, the pan-tilt may rotate too fast, causing the target to quickly move out of the field of view. Once the target is out of the field of view, it may be difficult for the system to find the target again. Sometimes when the pan-tilt rotates too fast to track the target, motion blur may occur, resulting in a decrease in image quality.

[0017] On the other hand, when performing a single PTZ inspection, it is necessary to continuously monitor specific devices such as electric meters, transformers, or electrical cabinets. However, when simply inspecting a certain target (such as an electric meter waiting to be tested device), it is often impossible to accurately determine its specific identification, especially when multiple similar devices exist together. It is even more difficult to distinguish the differences between them and it is impossible to distinguish which device has a problem. This problem will affect the accuracy and efficiency of the inspection, resulting in the failure to timely discover potential equipment failures. Currently, the commonly used methods for distinguishing devices include QR code or barcode recognition, wireless radio frequency identification (RFID) technology, environmental feature recording, and global positioning system (GPS). First, QR code or barcode recognition attaches a unique identifier to the device, so that the inspection equipment can quickly scan and obtain the device identity information. In addition, RFID technology uses radio frequency identification tags, so that the inspection personnel can automatically read its information when approaching the device. Environmental feature recording is also very important. It enhances the accuracy of identification by recording the device configuration and relative position around the device. Finally, GPS positioning can confirm the installation location of the device in real time, thereby further improving the recognition efficiency. However, each of them also has some disadvantages. QR code recognition requires an additional QR code scanning function. When affected by contamination or occlusion, the recognition rate may be reduced, and the management and maintenance of QR codes also increase the workload. Although RFID technology can be used for rapid identification, it is necessary to install RFID tags on the equipment and equip them with dedicated card readers, which increases the equipment and implementation costs. At the same time, the signal may fail to read in an interference environment. In addition, the accuracy of GPS positioning is affected by environmental factors, and the signal is unstable in some areas, requiring additional hardware support. The shortcomings of these technologies need to be considered in actual applications to ensure an effective inspection process.

[0018] On the other hand, with the continuous development of image processing technology, gimbal cameras have been widely used in monitoring, security, drone photography and other fields. However, in actual use environments, changes in lighting conditions usually have a significant impact on image quality. Under different lighting conditions such as strong sunlight, shadows or night, the images captured by the gimbal camera may be overexposed or underexposed, resulting in an imbalance in the contrast between light and dark, thereby affecting the visibility of details and the overall effect. When dealing with changes in lighting, the image processing technology in the related art often cannot effectively optimize the brightness and contrast of the image, resulting in the loss of image details and affecting the accuracy of subsequent analysis and recognition.

[0019] In summary, in the process of image acquisition through equipment inspection in the related technology, there are problems in the robot gimbal speed adjustment, image self-processing, specific positioning of the equipment, etc. Specifically, in terms of robot gimbal speed adjustment, when the target moves too fast, the gimbal may not be able to rotate in time, so it cannot lock the target; and when the target moves slowly, the gimbal may rotate too fast, causing the target to move out of the field of view quickly. In terms of image self-processing, most of the current algorithms for optimizing image quality are based on the brightness of the image itself, but if the brightness in the image is unevenly distributed, it may cause incorrect adjustments. For example, images with high contrast or obstructions may cause unreasonable overall brightness adjustments, affecting image quality. Simultaneous processing of images to automatically adjust brightness requires computing time, which may cause delays, especially when processing high-resolution images. When the ambient lighting changes rapidly (such as suddenly entering a brightly lit area), adjustments based on image processing may not be sensitive enough and cannot respond in time. In terms of specific equipment positioning, the current related technologies usually use QR code image recognition or GPS positioning to accurately obtain the identifier ID and location of the equipment (such as the electric meter). The method based on QR code image recognition is too cumbersome and requires other auxiliary identification (QR code, barcode or characters, etc.), which is easy to fall off and need to be replaced. Although RFID technology can be used for rapid identification, it is necessary to install RFID tags on the equipment and equip it with a dedicated card reader, which increases the equipment and implementation costs. At the same time, the signal may fail to read in an interference environment. The accuracy of GPS positioning is affected by environmental factors, and the signal is unstable in some areas, requiring additional hardware support. At the same time, if GPS positioning is used, it is difficult to distinguish between devices that are close to each other, resulting in errors in the returned data, which in turn leads to low inspection efficiency and poor reliability.

[0020] According to an embodiment of the present invention, a method embodiment of patrol control is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] Figure 1 is a flow chart of a patrol control method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0022] Step S102, based on the pan-tilt camera carried by the inspection vehicle, multiple image acquisition and image analysis are performed on the same relatively moving target to obtain multiple detection results, wherein the relatively moving target is used to indicate the target that is in relative motion with the inspection vehicle;

[0023] Optionally, multiple detection results are obtained by imaging the same relatively moving target with a pan-tilt camera carried on the inspection vehicle, obtaining multiple initial images, and analyzing the multiple initial images. The pan-tilt camera is a camera set on the pan-tilt carried on the inspection vehicle. The pan-tilt camera not only collects images of the same relatively moving target, but also collects details of its surrounding environment, providing richer data for subsequent feature extraction, which is conducive to improving the accuracy and robustness of target recognition. The initial image containing the same relatively moving target and its neighborhood environment can be used for subsequent auxiliary positioning of the same relatively moving target based on environmental features, which is particularly critical in scenes where similar targets are dense or the logo is difficult to read, and can significantly improve the specific positioning and recognition speed of the target. In addition, the addition of environmental information makes the collected target features more comprehensive, thereby avoiding the limitations of relying solely on the target appearance or logo for identification, especially when the target appearance is similar or the logo is damaged, the environmental features become the key information to distinguish different targets.

[0024] Optionally, the same relative motion target can be a key device focused on the fields of power, industry and infrastructure maintenance, and can be a stationary device with a fixed position or a moving device with a non-fixed position. For example, it can include, but is not limited to, electric meters, transformers, electrical cabinets (i.e., cabinets containing various electrical components and equipment, used to control and protect circuits in power systems), industrial equipment (such as production line equipment, automation devices, sensor nodes, etc. in factories), etc. installed in fixed positions, or it can be a device to be tested that is placed on a conveyor belt and moves with the conveyor belt. The same relative motion target can also be a staff member performing work activities in a target area (such as a factory building).

[0025] It should be noted that the same relative motion target is a target that is in relative motion with the inspection vehicle. The relative motion can be divided into two situations, one of which is the relative motion target when the inspection vehicle is stationary and the target is moving, and the other is the relative motion target when the inspection vehicle is moving and the target is stationary. When the inspection vehicle is stationary and the target is moving, the relative motion target can be: a worker in a moving state, a device or component to be tested on a conveyor belt, etc. When the inspection vehicle is moving and the target is stationary, the relative motion target can be: an electric meter, a high-temperature cable, and other stationary equipment.

[0026] In an optional embodiment, a pan-tilt camera mounted on an inspection vehicle is used to perform multiple image acquisition and image analysis on the same relatively moving target to obtain multiple detection results, including: during the movement of the inspection vehicle, based on the pan-tilt camera mounted on the inspection vehicle, an initial image of the same relatively moving target is acquired, wherein the initial image includes the same relatively moving target and environmental information within a predetermined neighborhood of the same relatively moving target; the brightness of the initial image is adjusted to obtain a target image; target detection is performed on the target image to obtain any detection result; and detection results corresponding to multiple target images are obtained in a manner to obtain any detection result.

[0027] Optionally, by adjusting the brightness of each initial image, the problem of overexposure or underexposure of the image caused by changing lighting conditions can be improved, thereby improving the overall clarity and detail of the image. This is crucial for subsequent feature extraction and target recognition, ensuring that the algorithm can obtain the most accurate information when analyzing the image. In addition, the target image obtained after adjusting the brightness has a more reasonable contrast and color distribution, which helps to identify the target more accurately in the future, so as to better distinguish the characteristics of the relatively moving target from the surrounding environment, especially in scenes with poor or changing lighting conditions. This preprocessing step can significantly improve the robustness of relatively moving target recognition. In addition, optimizing the brightness in the early stage of image processing can reduce feature extraction errors caused by poor image quality, thereby reducing false positives and false negatives in relatively moving target recognition, and improving the efficiency and reliability of inspection work. Through brightness adjustment, the images collected by the gimbal camera can better adapt to various lighting environments, whether in strong sunlight, in the shadow or at night, and can provide high-quality image input, so that target recognition can maintain consistent performance at different times and locations.

[0028] Through target detection, the features of the equipment in the image captured by the PTZ camera can be accurately extracted. For example, when the target to be identified in the image is a device, the corresponding target features may include, but are not limited to, key details such as the shape, size, color, and texture of the device; when the target to be identified in the image is a staff member, the corresponding target features may include, but are not limited to, the staff member's body shape information (contour information), facial image information, etc. At the same time, environmental features, such as objects, backgrounds, and positional relationships around the relatively moving target, are also detected. These environmental features help to more comprehensively understand the context in which the target is located, so that when the target features are similar, different targets can be distinguished by environmental differences to improve the accuracy and robustness of target recognition. The environmental features in the detection results can provide rich environmental perception information, which not only helps to locate the specific location of the target, but also helps to understand the target's operating environment, such as whether the target is in a normal working environment and whether there are potential fault hazards. Through the analysis of environmental features, the state of the target can be better judged, providing a basis for subsequent maintenance and management. The detection results obtained by the detection include target features and environmental features, which can be used as "keywords" for subsequent database queries. Through target detection, target features can be directly extracted and identified from images, reducing reliance on auxiliary identification methods such as QR codes, barcodes or RFID, and reducing the implementation and maintenance costs of the inspection system. At the same time, this method is more suitable for complex environments where identification may be missing or blocked, thereby improving the flexibility and adaptability of target recognition.

[0029] Optionally, the target image can be processed by but is not limited to a target detection algorithm to detect the target in the image and obtain a detection result, wherein the target detection algorithm can include but is not limited to You Only LookOnce (YOLO), Single Shot MultiBox Detector (SSD), Faster Region-based Convolutional Neural Network (Faster R-CNN), etc.

[0030] In an optional embodiment, the brightness of the initial image is adjusted to obtain a target image, including: based on the initial image, using an image adjustment model to obtain a target gamma value, wherein the image adjustment model pre-learns the correspondence between the image and the gamma value; based on the target gamma value, the brightness of the initial image is adjusted to obtain the target image.

[0031] Optionally, an image adjustment model is used to automatically determine the target gamma value to optimize the brightness of the initial image. This process is based on deep learning technology and draws on the human eye's adaptation mechanism to different lighting conditions to improve the efficiency and accuracy of image processing. In the image processing process of relatively moving target inspection, the initial image captured by the gimbal camera may be affected by complex lighting conditions, such as high-intensity sunlight, shadows, or insufficient light at night. These lighting conditions will directly affect the brightness and contrast of the image, and thus affect the accuracy of subsequent feature extraction and recognition. Therefore, the target gamma value is automatically determined by using a deep learning image adjustment model to adapt to different lighting environments.

[0032] Optionally, the image adjustment model is obtained by training a large amount of image data containing different lighting conditions. During the training process, the model learns the correspondence between image features and gamma values, and can predict a gamma value that is most suitable for improving the visual effect of the image based on the brightness distribution, color information and ambient lighting conditions of the input image. This prediction process is automated and does not require manual intervention, thereby significantly improving the efficiency and adaptability of image processing. After predicting the target gamma value based on the image adjustment model, this target gamma value will be applied to perform nonlinear brightness adjustment on the initial image. The adjustment of the gamma value can follow the formula Iout=Iin^γ, where Iout is the brightness value of the output image, Iin is the brightness value of the input image, and γ is the gamma value.

[0033] By adjusting the gamma value, for example, when the gamma value is less than 1, the details of the dark part of the image can be enhanced, making the shadow part richer; on the contrary, when the gamma value is greater than 1, the details of the bright part can be enhanced, making the highlight area brighter and clearer. It can effectively improve the brightness distribution in the image, enhance the visibility of the dark details, and control the overexposure of the bright part, so that the overall image is closer to the natural perception of the human eye. The target image adjusted by gamma not only has a more balanced brightness and a more reasonable contrast, but also can better retain and highlight the detailed features of the relatively moving target and the environment. This is crucial for subsequent feature extraction and target recognition, thereby ensuring that high recognition accuracy and efficiency can be maintained under various lighting conditions.

[0034] It should be noted that in actual use environments, changes in lighting conditions usually have a significant impact on image quality. Under different lighting conditions such as strong sunlight, shadows or nighttime, the images captured by the gimbal camera may be overexposed or underexposed, resulting in an imbalance in the contrast between light and dark, which affects the visibility of details and the overall effect. In order to solve these problems, it is particularly important to learn from the adaptability of the human eye to light and the gamma adjustment algorithm. The human eye can self-adjust in complex and changing lighting conditions and adapt to changes in light and dark in real time, thereby maintaining visual balance and delicacy. The gamma adjustment algorithm is a nonlinear image enhancement technology that can effectively improve the brightness and contrast of the image by adjusting the gamma value of the image, making the dark details more visible while controlling the overexposure of the bright parts. When processing images, this technology can improve the image quality in different environments by reasonably balancing the lighting and detail performance. However, the gamma value of many algorithms is directly obtained based on the brightness and darkness of the image itself, which may cause inaccurate brightness adjustment. In this embodiment, by introducing an image adjustment model to automatically predict the gamma value and adjust the brightness of the initial image, the problem of unstable image quality caused by changes in lighting conditions during the inspection process can be solved. This technology combines the predictive ability of deep learning with the physiological characteristics of the human eye's adaptation to light, which can significantly improve the level of intelligent image processing and provide strong technical support for more accurate and efficient identification and positioning of relatively moving targets. By optimizing image quality, target feature extraction and identity confirmation tasks can be performed more stably, thereby improving the performance of the entire inspection and monitoring process.

[0035] In an optional embodiment, before obtaining the target gamma value based on the initial image using the image adjustment model, the method also includes: acquiring multiple images, wherein the multiple images correspond to different light intensities; determining multiple gamma values ​​corresponding to the multiple images, wherein the multiple images correspond one-to-one to the multiple gamma values; and training the initial model based on the multiple images and the multiple gamma values ​​to obtain the image adjustment model.

[0036] Optionally, in the early stages of training the image conditioning model, the gimbal camera will collect a series of images that can cover the range of light intensities from very dark to very bright, so that the characteristics of images under various lighting conditions can be fully understood, providing diverse data support for model training. By collecting these images, the impact of lighting changes on image brightness, contrast, and color can be captured, laying the foundation for subsequent model training. After acquiring multiple images with different light intensities, the next task is to find an optimal gamma value for each image. The choice of gamma value directly affects the brightness and contrast of the image, which in turn affects the visual effect of the image and the accuracy of subsequent processing. A most suitable gamma value can be determined for each image through a series of tests or preset adjustment algorithms to ensure that the image can maintain good visual quality and detail visibility even under extreme lighting conditions. This process is the core of image preprocessing and helps to improve the robustness of image recognition. With images of different light intensities and their corresponding gamma values, the next step is to use this data to train the initial model to build the final image conditioning model. The training process usually involves deep learning techniques such as Convolutional Neural Network (CNN), where the model learns how to automatically predict a gamma value based on the image's features, thereby adjusting the brightness of any new input image.

[0037] Optionally, during the training process, the model takes multiple images and corresponding gamma values ​​as input, and adjusts the model parameters through back propagation and loss function optimization so that it can accurately predict the gamma value suitable for the input image. This training phase requires a lot of computing resources and time, but the resulting image adjustment model will be able to perform gamma adjustment on images in real time and efficiently during the inspection process, significantly improving the intelligence and adaptability of image processing.

[0038] For example, when shooting with the same gimbal, 100 images with different light intensities are collected. The collection of these images covers a variety of lighting conditions from extremely dark to extremely bright to ensure that comprehensive lighting change data can be obtained. Subsequently, a specific gamma value is applied to each image through the gamma adjustment algorithm to optimize its brightness for the best effect. This process not only improves the visual quality of the image, but also enhances the visibility of details in the image. In order to further improve processing efficiency and accuracy, a model based on the relationship between image features and gamma values ​​can be generated through training. This model enables the automatic identification of the lighting conditions of new images when they are collected in subsequent image processing, and the rapid application of appropriate gamma values ​​for adjustment.

[0039] Through the above training process, it can be ensured that the image adjustment model can effectively cope with various lighting conditions, improve image quality by automatically adjusting the gamma value, and provide higher quality image input for subsequent target detection and recognition. It can not only improve the degree of automation of image processing, but also reduce the dependence on manual adjustment, making the inspection system more stable and reliable in complex environments. Through the introduction of deep learning technology, the model can learn the complex correspondence between lighting and gamma values ​​from the training data, thereby further improving the intelligent level of image processing, which is an important technical support for building an efficient target inspection system.

[0040] In an optional embodiment, after performing target detection on the target image and obtaining the detection result of the target image, the method also includes: detecting whether there is a reference image matching the detection result of the target image in the images stored in the database; if the reference image is stored in the database, determining a reference identifier corresponding to the reference image; and based on the reference identifier, storing the target image to a corresponding position in the database.

[0041] Optionally, after target detection is performed on the target image, feature matching is performed between the obtained detection result and the image stored in the database to find out whether there is a reference image with a high degree of similarity. This process utilizes the target features and environmental features in the detection results. By comparison, it can be confirmed whether the currently detected relative motion target is a known target. If a reference image matching the detection result of the current target image is found in the database, the identifier (reference identifier) ​​corresponding to the reference image is determined. Based on this identifier, the current target image and its detection result are stored in the database at a location associated with the identifier, and an image file of the relative motion target is established or updated. Among them, when the relative motion target is a device, the corresponding identifier can be a device identifier; when the relative motion target is a person, the corresponding identifier can be a person identifier (such as a staff code).

[0042] In the above method, by querying the reference image in the database, the target can be identified more quickly, avoiding the complex process of re-analyzing and identifying the target each time the target is detected, thereby significantly improving the speed and efficiency of target identification. For known targets, their identification is confirmed by feature matching, thereby reducing repeated detection of the same target, avoiding the redundancy of storing similar data in the database, saving storage space, and simplifying data management. Associating and storing the detection results with the identification helps to build a historical state record of the target and realize continuous tracking of the target state. In the case where the relative motion target is the device to be tested, the above method has important value in target maintenance, state monitoring and data analysis, and helps to timely discover equipment abnormalities, perform fault prediction and preventive maintenance. In the case where the relative motion target is a person, the above method can be performed by performing feature analysis on the relative motion target, such as analysis of facial image information, action contour information, etc., thereby realizing the identity recognition of the target, operation specification recognition, etc., thereby improving the safety of the operation. By matching the detection results with the historical records in the database, efficient use and management of data can be achieved, providing more accurate and detailed data support for inspection tasks, and reducing the complexity of data maintenance.

[0043] Optionally, a feature matching algorithm can be used to detect whether there is a reference image matching the detection result in the images stored in the database, and a large amount of historical data can be analyzed in a short period of time to quickly determine the identification, thereby avoiding the inefficient method of checking targets one by one or relying on manual recognition in related technologies, and improving the degree of automation and efficiency of inspections.

[0044] Optionally, when the identification is recognized, intelligent analysis can also be performed based on the target type, historical status and current characteristics. For example, when the relative motion target is a device, the health status of the relative motion target can be predicted, maintenance needs can be evaluated, etc.; when the relative motion target is a staff member, the operating behavior, health status, etc. of the relative motion target can be predicted, thereby supporting more intelligent equipment management and action management.

[0045] Optionally, when the relative motion target is a device, the reference identifier may carry information such as the device type, device model, device number, etc., and may also carry the device installation location information. In this case, the target image may be stored in a corresponding position in the database based on the reference identifier. When the reference identifier does not include the device location information, that is, the device location information exists independently of the reference identifier, the location information may also be obtained through database matching, and the target image may be stored in a corresponding position in the database based on the reference identifier and the location information.

[0046] In an optional embodiment, detecting whether there is a reference image matching the detection result of the target image among the images stored in the database includes: calculating the similarity between the detection results of the target image and a plurality of groups of historical images stored in the database, wherein each group of historical images corresponds to an identifier; detecting whether there is a group of images having a similarity greater than a predetermined similarity threshold among the plurality of groups of historical images; if there is a group of images having a similarity greater than a predetermined similarity threshold among the plurality of groups of historical images, determining that the reference image is stored in the database, wherein the group of images having a similarity greater than the predetermined similarity threshold is the reference image; if there is no group of images having a similarity greater than the predetermined similarity threshold among the plurality of groups of historical images, determining that the reference image is not stored in the database.

[0047] Optionally, after each target inspection, the current image of the same relatively moving target (target image) will be obtained and compared with all historical images stored in the database to calculate the similarity. The purpose is to find the image in the database that is closest to the current target image, that is, the reference image. The calculation of similarity can be, but is not limited to, using methods such as Euclidean distance, cosine similarity or Hamming distance, depending on the type of feature vector, system design, and user habits. In order to determine whether the historical image is similar enough to the target image, a predetermined similarity threshold is set. This threshold can be determined by the requirements for the accuracy of feature recognition of relatively moving targets and the analysis of historical data. If the calculated similarity is higher than this threshold, it means that the historical image and the target image are similar enough in features and can be determined to be a match. On the contrary, if the similarity of all historical images to the target image is lower than the threshold, it is considered that no matching image is found. When it is detected that the similarity of one or more groups of historical images exceeds the predetermined threshold, these historical images will be regarded as reference images. The existence of reference images means that the same or very similar targets have been encountered before, and the identification of the reference images can be used to quickly locate the current target. This process can ensure the continuity of the target and the traceability of historical records. If the similarity between all historical images and the target image is lower than the threshold, it is determined that the current target or target state is unique and there is no matching reference image stored in the database. In this case, a new target identification can be created, and the target image and corresponding target information can be stored in the database as a reference for future identification. In this way, the relative moving target can be accurately identified and located while maintaining the update of the database and the integrity of the target historical data. By setting a reasonable similarity threshold, the relevant information of the previously stored target can be quickly retrieved during the inspection process, improving the inspection efficiency and reliability.

[0048] Optionally, each group of historical images includes one or more images, and the similarities between the multiple images included in each group of historical images and the target image can be calculated, or representative images can be selected from the multiple images included in each group of images to calculate the similarity between the representative images and the target image.

[0049] In an optional embodiment, the method further includes: generating a target identification corresponding to the target image when no reference image is stored in the database; and storing the target image and the target identification in the database in correspondence.

[0050] Optionally, when the inspection system compares the image (target image) captured by the PTZ camera with all the historical images (reference images) in the database, if the similarity between any historical image and the target image exceeds a preset threshold, this means that the currently detected relative motion target has not been recorded in the database and is a newly encountered target. At this time, a new identifier (target identifier) ​​is generated to uniquely identify the new target. The identifier can be generated based on a preset rule or algorithm, for example, it can be a combination of timestamp, target type and location information to ensure that each target has a unique identifier. After the target identifier is generated, a storage operation is performed to store the target identifier in the database in correspondence with the target image and its related metadata (such as the type, location, detection time, etc. of the target). This storage process not only includes the preservation of image data, but also involves the association of the identifier with the image information, ensuring that the image record of the relative motion target can be quickly retrieved based on the identifier in the future. When storing, a new database entry can also be created, or a new record can be added under an existing target type to keep the data structured and organized.

[0051] Through the above methods, it is possible to dynamically adapt to the addition of new targets, maintain real-time updates of the database and comprehensive records of target status. The generation and storage of new identifiers can ensure effective management even when facing unknown targets, and provide the necessary information basis for subsequent relative motion target identification, status monitoring and maintenance.

[0052] In an optional embodiment, the target image and the target identification are stored correspondingly in a database, including: based on a gimbal camera, collecting other images of the same relatively moving target, wherein the other images are images of the same relatively moving target collected along a predetermined shooting direction; performing feature extraction on the target image and the other images respectively to obtain feature extraction results; and storing the feature extraction results, the target image, the other images and the target identification correspondingly in a database.

[0053] Optionally, when a new target is detected, in addition to the initially captured target image, the PTZ camera can also shoot the relatively moving target along multiple predetermined directions or angles to collect more comprehensive image data. These "other images" cover different perspectives of the target, which helps to more comprehensively understand the appearance characteristics of the target. For example, for a relatively moving target such as an electric meter, the PTZ camera may shoot from multiple angles such as the front, side, and oblique view to obtain the entire image as well as the front and rear camera images, thereby ensuring that the collected image can fully reflect the target's identification information, status details, and surrounding environment. The newly generated target identification is further stored in the database together with all the collected images of the relatively moving target (including the initial target image and other images taken subsequently). When storing, each image is associated with the target identification to ensure that all image data can be correctly indexed and retrieved. This operation not only includes the preservation of image files, but also involves the association of identification with image metadata, such as shooting time, location information, angle information, etc., to maintain data integrity.

[0054] In this embodiment, by storing multi-angle target images, a richer target feature library can be constructed, which can provide multi-view references in the subsequent relative motion target recognition process, which helps to improve the accuracy and robustness of recognition. For example, if at some point in the future, the angle or position of the relative motion target is different from that at the time of initial recognition, it is still possible to find a matching identifier by retrieving the multi-angle images stored in the database, thereby ensuring continuous tracking and management of the target state. The above method can not only ensure the complete record of new target information, but also enhance the description ability of target features through the collection of multi-angle images, and provide powerful data support for subsequent relative motion target recognition, state monitoring and fault diagnosis, so that targets (such as equipment, staff) can be managed more accurately and reliably in a complex and changing environment.

[0055] Step S104, using an optical flow tracking method to perform optical flow analysis on multiple detection results to obtain the movement speed and position of the same relative moving target in the field of view, wherein the field of view is the collection field of view of the preset pan-tilt camera;

[0056] Optionally, the optical flow tracking method is used to analyze the motion trend of pixels in a sequence of images to determine the target's motion direction and speed. In a series of multiple target images (target image sequence), optical flow analysis is used to detect the motion changes of the results (such as electric meters, transformers, staff, etc.), and then the relative motion speed of the target and its position in the field of view of the gimbal camera are calculated. This analysis process can capture the position changes of the target at different time points, providing key speed information for subsequent tracking strategies.

[0057] Step S106, determining the total rotation angle of the inspection vehicle and the PTZ camera based on the movement speed and the position in the field of view, wherein the total rotation angle is the total angle that the inspection vehicle and the PTZ camera need to rotate when keeping the same relative moving target at the center of the field of view;

[0058] Optionally, the total rotation angle can be obtained by calculation or table lookup based on the target's movement speed and position in the field of view. For example, a comparison table can be pre-set in which the correspondence between the target's movement speed interval, position interval in the field of view, and rotation angle interval is pre-stored. By looking up the table, the total angle that the inspection vehicle and the pan-tilt camera need to rotate to keep the same relatively moving target at the center of the field of view can be quickly determined. In the above manner, when determining the total rotation angle of the inspection vehicle and the pan-tilt, not only the movement speed of the target is taken into account, but also the current position of the target in the field of view of the pan-tilt camera. The total angle that the inspection vehicle and the pan-tilt camera need to rotate (i.e., the total rotation angle) obtained on this basis is also more accurate and reliable.

[0059] Step S108, determining the rotation angles corresponding to the inspection vehicle and the PTZ camera respectively based on the total rotation angle and the speed ratio of the two, wherein the speed ratio of the two is the speed ratio between the inspection vehicle and the PTZ camera;

[0060] Optionally, determine the proportional relationship between the angle adjustment of the inspection vehicle and the PTZ camera (i.e., the speed ratio of the two), which is based on the movement speed obtained from the optical flow analysis and the simulation of the coordinated movement characteristics of the human eye and neck. The proportional relationship ensures that the adjustment of the PTZ camera and the inspection vehicle is both complementary and coordinated, so that the target is always kept in the center of the field of view, and the tracking stability and accuracy can be maintained even when the target moves quickly or changes direction suddenly.

[0061] Specifically, the speed ratio of the two can be determined based on the principle of coordinated movement of the human eye and neck. The speed ratio of the two can ensure the rationality of the rotation speed and angle distribution of the two to achieve efficient tracking of fast-moving targets. For example, if the target is moving quickly to the left, the inspection vehicle may be required to accelerate to the left, while the PTZ camera makes more precise angle adjustments to ensure the clarity and stability of the target image. After obtaining the target's movement speed information and the preset angle adjustment ratio, it is possible to calculate the angles that the inspection vehicle and the PTZ camera should each rotate when tracking the target.

[0062] Optionally, the angles that the inspection vehicle and the PTZ camera need to rotate are calculated based on their total rotation angle and the speed ratio of the two. This calculation takes into account the relative position and movement of the two in space, as well as the speed information of the target, to ensure smooth and accurate adjustment.

[0063] Specifically, if the target moves quickly, the inspection vehicle may need to turn a larger angle to compensate for the movement, while the PTZ camera may make fine adjustments at a smaller angle; conversely, if the target moves slowly, the PTZ camera can make larger angle adjustments to keep the target in the center of the field of view. This dynamic angle adjustment strategy ensures that no matter how the target moves, it can quickly respond and adjust the viewing angle to achieve continuous and clear target tracking. Finally, according to the calculated rotation angle, the steering of the inspection vehicle and the rotation of the PTZ camera can be controlled to ensure that the target is always in the center of the monitoring field of view. This control process may involve real-time calculation and feedback to adapt to the dynamic changes of the target movement. By synchronously adjusting the angles of the inspection vehicle and the PTZ camera, the limitations of single target adjustment can be overcome, and a wider range and more efficient target tracking can be achieved, especially when the target moves quickly, which can significantly improve the accuracy and stability of tracking.

[0064] Step S110, controlling the inspection vehicle and the pan-tilt camera to rotate based on the rotation angles corresponding to the inspection vehicle and the pan-tilt camera respectively.

[0065] Optionally, based on the corresponding rotation angles of the inspection vehicle and the pan-tilt camera, corresponding control instructions are generated and issued to control the inspection vehicle and the pan-tilt camera to rotate accordingly, so that the relatively moving target to be inspected remains in the center of the field of view, thereby achieving effective tracking and monitoring.

[0066] It should be noted that optical flow tracking is an image processing technology that constructs an optical flow field by analyzing pixel changes in continuous image frames. This optical flow field can be regarded as a vector field, in which the optical flow vector of each pixel reflects the moving direction and speed of the pixel in the image sequence over time. By calculating and analyzing the optical flow field, the motion state of the target in the image can be effectively captured and determined. For example, when an object moves in the picture, the moving direction and relative motion speed information of the object can be obtained through optical flow analysis. After obtaining the target's motion speed information in the image, the next step is to associate these motion parameters with the physiological motion characteristics of the human eye and neck. Specifically, the pan-tilt head is equivalent to the human eye in this system, and the inspection car is similar to the neck. In order to ensure that the pan-tilt head camera can track the target quickly and accurately, it is necessary to allocate the rotation angle of the pan-tilt head and the inspection car according to the proportional relationship between the rotation angles of the human eye and the neck. This proportional relationship involves not only their rotation angles, but also the corresponding rotation angular velocity. By calculating the rotation speed of the PTZ and the inspection vehicle, the speed ratio of the two can usually be set to a constant value, and then the rotation angle between the two devices can be reasonably allocated according to the previously determined target movement speed. This allocation follows the natural coordination law of eye and neck movement, ensuring that the rotation of the PTZ and the inspection vehicle is synchronized, thereby ensuring that the PTZ camera quickly aims at the target and responds. This rotation control scheme based on the human eye mechanism can not only improve the response speed of the PTZ, but also improve the accuracy of target tracking. It is an effective application of intelligent monitoring technology.

[0067] In the above method, through optical flow tracking and dynamic angle adjustment strategy, the angle of the inspection vehicle and the gimbal camera can be adjusted in real time to ensure continuous tracking of the target in a complex environment, which plays an important role in improving inspection efficiency and monitoring quality.

[0068] Optionally, a feature extraction algorithm may be used, but is not limited to, to obtain unique features of each monitored target in the target image and other images to obtain a feature extraction result, wherein the feature extraction algorithm may include, but is not limited to, Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), or a feature extraction model based on deep learning. These features can greatly distinguish different targets, such as shape, color, texture, and other important information. These features are converted into feature vectors in preparation for subsequent storage and matching.

[0069] It is still necessary to explain that in the process of target tracking based on the robot gimbal, the adjustment of the rotation speed is a key issue. There is no accurate estimation of the speed of the tracked object, resulting in a faster or slower tracking speed of the gimbal. When the target moves too fast, the gimbal (i.e., the gimbal camera) may not be able to rotate in time, and thus cannot lock the target; when the target moves slowly, the gimbal may rotate too fast, causing the target to quickly move out of the field of view. Once the target is out of the field of view, it may be difficult for the system to find the target again. Sometimes when the gimbal rotates too fast to track the target, motion blur may occur, resulting in a decrease in image quality. Therefore, in order to solve this problem, an intelligent speed adjustment strategy based on the optical flow tracking method is proposed by drawing on the principles of the human visual system. The optical flow tracking method can effectively simulate the brain's estimation of the target speed. By analyzing the movement changes of the target in the field of view, the target's speed information can be captured in real time. This information will be used to dynamically adjust the rotation speed of the gimbal to ensure that the target always remains in the center of the field of view. In addition, in view of the complexity of the target movement, a tracking scheme combining the joint movement of the gimbal and the robot body is proposed. This solution imitates the coordination mechanism of human eyes and neck, and increases the angle of the pan / tilt rotation through the movement of the robot body, thereby improving tracking efficiency. Specifically, when the target moves quickly, the robot body can accelerate toward the target, providing greater flexibility; when the target moves slowly, the pan / tilt can be finely adjusted to ensure that the target does not quickly leave the field of view.

[0070] Through the above steps S102 to S108, the optical flow tracking method is used to track and locate the target in the image, and the inspection vehicle and the pan-tilt camera angle are accurately adjusted synchronously based on the determined movement speed, thereby improving the adjustment accuracy of the robot pan-tilt, and then improving the image acquisition quality and image reliability of the robot pan-tilt, avoiding the technical effect of target loss, and then solving the technical problem of target loss and poor image reliability caused by the excessively fast relative movement speed between the inspection vehicle and the target during the inspection process.

[0071] Based on the above embodiment and optional embodiment, the present invention proposes an optional implementation of a patrol control method, the method comprising:

[0072] S1, light adjustment based on human eyes: In the process of shooting based on the PTZ camera installed on the inspection vehicle, in order to improve the overall quality and detail performance of the image, the sensitivity of the human eye to light can be used to process the captured image through the gamma adjustment algorithm. This process is inspired by the physiological characteristics of the human eye, that is, the pupil can automatically adjust its opening and closing according to the change of light intensity, thereby optimizing the quality of light incident on the retina. Under different lighting conditions, the perception of the human eye is nonlinear, and appropriate image processing can make the output effect more in line with human visual habits. Gamma adjustment is a widely used nonlinear image processing technology, mainly used to adjust the brightness and contrast of the image. The gamma value is the core parameter of this process. It is usually a positive number that affects the distribution of brightness in the image. The basic formula of gamma adjustment is: Iout = Iin^γ, where Iout represents the brightness value of the output image, Iin is the brightness value of the input image, and γ is the gamma value. By applying this formula to the image, its visual effect can be significantly affected. The brightness perception ability of the human eye under different lighting conditions is not linear. For example, in low-light environments, the human eye's sensitivity to brightness changes increases, which means that even subtle differences in brightness can be clearly perceived. Therefore, in the image processing process, we can select an appropriate gamma value to adjust according to this physiological characteristic to optimize the image's visual performance. When selecting a gamma value, generally speaking, when the gamma value is less than 1, the details of the dark part of the image can be enhanced, making the shadow part richer; on the contrary, when the gamma value is greater than 1, the details of the bright part can be enhanced, making the highlight area brighter and clearer. Therefore, it is particularly important to choose a reasonable gamma value under different image conditions. When shooting with the same gimbal, the gimbal camera collects 100 images with different light intensities. The collection of these images covers a variety of lighting conditions from extremely dark to extremely bright to ensure that comprehensive lighting change data can be obtained. Subsequently, a specific gamma value is applied to each image through a gamma adjustment algorithm to optimize its brightness for the best effect. This process not only improves the visual quality of the image, but also enhances the visibility of details in the image. In order to further improve processing efficiency and accuracy, a training based on 100 images and the gamma value of each image is performed to generate an image adjustment model based on the relationship between image features and gamma values. This model enables the system to automatically identify the lighting conditions of new images when they are captured in subsequent image processing, and quickly apply the appropriate gamma value for adjustment. The gamma-adjusted images can be input into the detection algorithm in a better state, ensuring that the subsequent analysis and recognition process is more accurate and efficient, thereby improving the overall monitoring and recognition performance.

[0073] S2, imitating human eye recognition and memory target: only identify the exact target through a single visual image, such as a room with three identical electricity meters, the human eye distinguishes between meter one, meter two and meter three through their surrounding features. The process of recording the detected target and saving it to the database for subsequent feature matching can be divided into the following detailed steps: First, the gimbal captures the image or video stream of the surrounding environment in real time. The captured image has a large field of view and includes as many features as possible. In this process, the target detection algorithm (such as YOLO, SSD or Faster R-CNN, etc.) is used to process the image and detect the target in the image. After the target is detected, it is first compared with the image in the database. If it is a previously detected target, the ID and location information of the previously detected target will be matched with the currently detected content and sent to the background. If the match with the database fails, a new target ID and target feature information are created and entered into the target library. At this time, the target feature information contains the entire image taken by the gimbal and the front and rear camera images to save sufficient feature information. The feature information uses feature extraction algorithms (such as SIFT, SURF, ORB, or feature extraction models based on deep learning) to obtain unique features for each detected target. These features can greatly distinguish different targets, such as shape, color, texture, and other important information. These features are converted into feature vectors in preparation for subsequent storage and matching.

[0074] Then, the system enters the data storage stage. In this stage, a database is designed (relational databases such as MySQL, or non-relational databases such as MongoDB, etc. can be used) to store these feature vectors and related information. Each record in the database not only stores the feature vector of the target, but also includes meta-information such as the target's ID, detection time, location, and corresponding image data. By inserting this information into the database, the system lays the foundation for subsequent feature matching. At the next detection, the system will recapture the video stream and detect the target, and extract features in the same way. At this time, the system will match the currently detected target based on the existing feature vector. In this stage, metric learning methods (such as cosine similarity or Euclidean distance) are used to evaluate the similarity between the current feature vector and the feature vector stored in the database. The system will retrieve historical records with high similarity to identify whether they are previously known targets. Finally, once the target is recognized again, the system will update the information in the database to reflect the current status and location to ensure the real-time and accuracy of the data. At the same time, the system will monitor the performance of the entire process, and provide feedback and optimization based on the detection and recognition results, and dynamically adjust the feature extraction and matching algorithms to improve the overall recognition effect. Through this series of processes, the pan-tilt human eye system can effectively record and save each detected target, providing strong support for subsequent feature matching and target recognition, thereby achieving more accurate and efficient monitoring tasks.

[0075] S3, tracking the target by imitating the human eye, and adaptively adjusting the angles of the inspection vehicle and the pan / tilt: The first step is to use the optical flow tracking method to simulate the brain's rough estimation of the target's movement speed. The optical flow tracking method is an image processing technology that constructs an optical flow field by analyzing the pixel changes in consecutive image frames. This optical flow field can be regarded as a vector field, in which the optical flow vector of each pixel reflects the movement direction and speed of the pixel over time in the image sequence. By calculating and analyzing the optical flow field, the system can effectively capture and determine the motion state of the target in the image. For example, when an object moves in the picture, the moving direction and relative movement speed information of the object can be obtained through optical flow analysis. After obtaining the movement speed information of the target in the image, the next step is to associate these motion parameters with the physiological movement characteristics of the human eye and neck. Specifically, the pan / tilt in this system is equivalent to the human eye, and the inspection vehicle is similar to the neck. In order to ensure that the pan / tilt camera can track the target quickly and accurately, it is necessary to allocate the rotation angles of the pan / tilt and the inspection vehicle according to the target proportional relationship between the rotation angles of the human eye and neck. This proportional relationship involves not only their rotation angles, but also the corresponding rotation angular velocity. By calculating the rotation speeds of the PTZ and the inspection vehicle, the speed ratio of the two can be set to a constant value, and then the rotation angles between the two devices can be reasonably allocated according to the previously determined target movement speed. This allocation follows the natural coordination law of eyeball and neck movement, ensuring that the rotation of the PTZ and the inspection vehicle is synchronized, thereby ensuring that the PTZ camera quickly aims at the target and responds. This rotation control scheme based on the human eye mechanism can not only improve the response speed of the PTZ, but also improve the accuracy of target tracking. It is an effective application of intelligent monitoring technology. In addition, this method also has good real-time performance, is suitable for use in complex and dynamic monitoring environments, can quickly adapt to changes in target movement, and promote the intelligence of inspection and monitoring work. Through this innovative design, the system can show great practical value in many practical applications.

[0076] This embodiment can achieve at least one of the following effects: 1) The rotation speed of the pan-tilt and the vehicle body itself is allocated according to the same ratio of the rotation angle allocation of the eyes and neck as the ratio of the rotation angular velocity of the eyes and neck to track the target faster. By simulating the rotation angle allocation of the eyes and neck to make the pan-tilt and the vehicle body rotate faster, the pan-tilt can follow the target more quickly to prevent the target from being lost. 2) For the same pan-tilt, it will collect 100 pictures of different light intensities, and adjust the picture brightness to a specific gamma value to achieve the optimal brightness. After training, a model of pictures and gamma values ​​is obtained. When processing pictures later, each picture will be gamma processed and then input into the detection algorithm. The gamma value obtained by deep learning can make the input image brightness more conducive to the algorithm to identify the target. 3) The ID of the detection target is identified based on the information features of the large-viewing field image, rather than based on QR code or GPS positioning identification, which reduces costs and makes identification and positioning more accurate. Through the matching of multiple features of the large-viewing field image, the ID and position of a certain electric meter or other detection target can be better located.

[0077] In this embodiment, a patrol control device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module" and "device" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0078] According to an embodiment of the present invention, a device embodiment for implementing the above inspection control method is also provided. Figure 2 is a schematic diagram of the structure of a patrol control device according to an embodiment of the present invention. Figure 2 As shown, the inspection control device includes: an acquisition module 200, a streamer analysis module 202, a first angle determination module 204, a second angle determination module 206, and a control module 208, wherein:

[0079] The acquisition module 200 is used to perform multiple image acquisition and image analysis on the same relatively moving target based on the pan-tilt camera carried by the inspection vehicle to obtain multiple detection results, wherein the relatively moving target is used to indicate the target that is in relative motion with the inspection vehicle;

[0080] The light flow analysis module 202 is used to perform light flow analysis on multiple detection results using an optical flow tracking method to obtain the motion speed and position of the same relative motion target in the field of view, wherein the field of view is the acquisition field of view of a preset pan-tilt camera;

[0081] The first angle determination module 204 is used to determine the total rotation angle of the inspection vehicle and the PTZ camera based on the motion speed and the position in the field of view, wherein the total rotation angle is the total angle that the inspection vehicle and the PTZ camera need to rotate to keep the same relative moving target at the center of the field of view;

[0082] The second angle determination module 206 is used to determine the rotation angles corresponding to the inspection vehicle and the PTZ camera respectively based on the total rotation angle and the speed ratio of the two, wherein the speed ratio of the two is the speed ratio between the inspection vehicle and the PTZ camera;

[0083] The control module 208 is used to control the rotation of the inspection vehicle and the pan-tilt camera based on the rotation angles corresponding to the inspection vehicle and the pan-tilt camera respectively.

[0084] In the embodiment of the present invention, an acquisition module 200 is set up to perform multiple image acquisition and image analysis on the same relatively moving target based on the pan-tilt camera carried by the inspection vehicle to obtain multiple detection results, wherein the relatively moving target is used to indicate the target that is in relative motion with the inspection vehicle; a light flow analysis module 202 is used to perform light flow analysis on multiple detection results using an optical flow tracking method to obtain the movement speed and position in the field of view of the same relatively moving target, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; a first angle determination module 204 is used to determine the total rotation angle of the inspection vehicle and the pan-tilt based on the movement speed and the position in the field of view, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center position of the field of view; a second angle determination module 204 is used to determine the total rotation angle of the inspection vehicle and the pan-tilt camera based on the movement speed and the position in the field of view, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center position of the field of view; Module 206 is used to determine the rotation angles corresponding to the inspection vehicle and the pan-tilt camera respectively based on the total rotation angle and the speed ratio of the two, wherein the speed ratio of the two is the speed ratio between the inspection vehicle and the pan-tilt camera; control module 208 is used to control the rotation of the inspection vehicle and the pan-tilt camera based on the rotation angles corresponding to the inspection vehicle and the pan-tilt camera respectively, thereby achieving the purpose of tracking and positioning the target in the image using the optical flow tracking method, and accurately adjusting the angles of the inspection vehicle and the pan-tilt camera synchronously based on the determined movement speed, thereby achieving the technical effect of improving the adjustment accuracy of the robot pan-tilt, and then improving the image acquisition quality and image reliability of the robot pan-tilt, and avoiding target loss, thereby solving the technical problem of target loss and poor image reliability caused by the excessively fast relative movement speed between the inspection vehicle and the target during the inspection process.

[0085] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0086] It should be noted that the acquisition module 200, the streamer analysis module 202, the first angle determination module 204, the second angle determination module 206, and the control module 208 correspond to steps S102 to S110 in the embodiment, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the modules as part of the device can be run in a computer terminal.

[0087] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0088] The above-mentioned inspection control device may also include a processor and a memory. The above-mentioned acquisition module 200, light stream analysis module 202, first angle determination module 204, second angle determination module 206, control module 208, etc. are all stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to realize corresponding functions.

[0089] The processor includes a kernel, which retrieves the corresponding program module from the memory. The kernel may be one or more. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.

[0090] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute any of the above inspection control methods.

[0091] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group, and the non-volatile storage medium includes a stored program.

[0092] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: based on the pan-tilt camera carried on the inspection vehicle, multiple image acquisition and image analysis are performed on the same relatively moving target to obtain multiple detection results, wherein the relative moving target is used to indicate the target that is in relative motion with the inspection vehicle; optical flow tracking method is used to perform optical flow analysis on multiple detection results to obtain the movement speed and position of the same relatively moving target in the field of view, wherein the field of view is the preset acquisition field of view of the pan-tilt camera; based on the movement speed and the position in the field of view, the total rotation angle of the inspection vehicle and the pan-tilt is determined, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center of the field of view; based on the total rotation angle and the speed ratio of the two, the corresponding rotation angles of the inspection vehicle and the pan-tilt camera are determined, wherein the speed ratio of the two is the speed ratio between the inspection vehicle and the pan-tilt camera; based on the corresponding rotation angles of the inspection vehicle and the pan-tilt camera, the rotation of the inspection vehicle and the pan-tilt camera is controlled.

[0093] According to an embodiment of the present application, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, wherein the program executes any one of the inspection control methods when it is run.

[0094] According to an embodiment of the present application, an embodiment of a computer program product is also provided. When executed on a data processing device, it is suitable for executing a program that initializes any one of the above-mentioned inspection control method steps.

[0095] Optionally, the above-mentioned computer program product, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: based on the pan-tilt camera carried on the inspection vehicle, multiple image acquisition and image analysis are performed on the same relatively moving target to obtain multiple detection results, wherein the relative moving target is used to indicate the target that is in relative motion with the inspection vehicle; optical flow tracking method is used to perform optical flow analysis on multiple detection results to obtain the movement speed and position of the same relatively moving target in the field of view, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; based on the movement speed and the position in the field of view, the total rotation angle of the inspection vehicle and the pan-tilt is determined, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center position of the field of view; based on the total rotation angle and the speed ratio of the inspection vehicle and the pan-tilt camera for angle adjustment, the rotation angles corresponding to the inspection vehicle and the pan-tilt camera are determined; based on the rotation angles corresponding to the inspection vehicle and the pan-tilt camera, the rotation of the inspection vehicle and the pan-tilt camera is controlled.

[0096] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: based on a pan-tilt camera carried on a patrol vehicle, multiple image acquisitions and image analyses are performed on the same relatively moving target to obtain multiple detection results, wherein the relatively moving target is used to indicate a target that is in relative motion with the patrol vehicle; an optical flow tracking method is used to perform optical flow analysis on the multiple detection results to obtain a movement speed and a position in a field of view of the same relatively moving target, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; based on the movement speed and the position in the field of view, a total rotation angle of the patrol vehicle and the pan-tilt is determined, wherein the total rotation angle is the total angle that the patrol vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center position of the field of view; based on the total rotation angle and the speed ratio of the two, the rotation angles corresponding to the patrol vehicle and the pan-tilt camera are determined; based on the rotation angles corresponding to the patrol vehicle and the pan-tilt camera, the rotation of the patrol vehicle and the pan-tilt camera is controlled.

[0097] The above sequence of the embodiments of the present invention is for description only and does not represent the superiority or inferiority of the embodiments.

[0098] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above modules can be a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, modules or indirect coupling or communication connection of modules, which can be electrical or other forms.

[0100] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0102] If the above-mentioned integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a non-volatile storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0103] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A patrol inspection control method, characterized in that: include: Based on the pan-tilt camera carried on the inspection vehicle, multiple image acquisition and image analysis are performed on the same relatively moving target to obtain multiple detection results, wherein the relatively moving target is used to indicate the target that is in relative motion with the inspection vehicle; Performing optical flow analysis on the multiple detection results using an optical flow tracking method to obtain the movement speed and position of the same relatively moving target in a field of view, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; Based on the movement speed and the position in the field of view, determining the total rotation angle of the inspection vehicle and the pan-tilt camera, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate to keep the same relatively moving target at the center of the field of view; Based on the total rotation angle and the speed ratio of the two, determining the rotation angles corresponding to the inspection vehicle and the pan-tilt camera respectively, wherein the speed ratio of the two is the speed ratio between the inspection vehicle and the pan-tilt camera; The inspection vehicle and the pan-tilt camera are controlled to rotate based on their respective corresponding rotation angles.

2. The method according to claim 1, characterized in that The PTZ camera mounted on the inspection vehicle performs multiple image acquisition and image analysis on the same relatively moving target to obtain multiple detection results, including: During the movement of the inspection vehicle, based on the pan-tilt camera carried by the inspection vehicle, an initial image of the same relatively moving target is collected, wherein the initial image includes the same relatively moving target and environmental information within a predetermined neighborhood of the same relatively moving target; Performing brightness adjustment on the initial image to obtain a target image; Performing target detection on the target image to obtain any detection result; The multiple test results are obtained by adopting the method of obtaining any one of the test results.

3. The method according to claim 2, characterized in that The step of adjusting the brightness of the initial image to obtain the target image includes: Based on the initial image, an image adjustment model is used to obtain a target gamma value, wherein the image adjustment model pre-learns a corresponding relationship between the image and the gamma value; Based on the target gamma value, the brightness of the initial image is adjusted to obtain the target image.

4. The method according to claim 3, characterized in that Before obtaining the target gamma value based on the initial image by using the image adjustment model, the method further includes: Acquire multiple images, wherein the multiple images correspond to different light intensities; Determining a plurality of gamma values ​​corresponding to the plurality of images, wherein the plurality of images correspond one-to-one to the plurality of gamma values; Based on the multiple images and the multiple gamma values, an initial model is trained to obtain the image adjustment model.

5. The method according to claim 2, characterized in that: After performing target detection on the target image to obtain the detection result of the target image, the method further includes: Check whether there is a reference image matching the detection result of the target image among the images stored in the database; In a case where the reference image is stored in the database, determining a reference identifier corresponding to the reference image; Based on the reference identifier, the target image is stored in a corresponding position in the database.

6. The method according to claim 5, characterized in that Whether there is a reference image matching the detection result of the target image among the images stored in the detection database includes: Calculating the similarity between the detection results of the target image and the multiple groups of historical images stored in the database, wherein each group of historical images corresponds to an identifier; Detecting whether there is a group of images having a similarity greater than a predetermined similarity threshold among the multiple groups of historical images; When there is a group of images whose similarity is greater than the predetermined similarity threshold among the multiple groups of historical images, determining that the reference image is stored in the database, wherein the group of images whose similarity is greater than the predetermined similarity threshold is the reference image; If there is no group of images whose similarity is greater than the predetermined similarity threshold among the multiple groups of historical images, it is determined that the reference image is not stored in the database.

7. The method according to claim 5, characterized in that The method further comprises: In the case where the reference image is not stored in the database, generating a target identifier corresponding to the target image; The target image and the target identification are stored in the database in correspondence.

8. The method according to claim 7, characterized in that The storing the target image and the target identification in the database in correspondence includes: Based on the gimbal camera, collecting other images of the same relatively moving target, wherein the other images are images of the same relatively moving target collected along a predetermined shooting direction; Performing feature extraction on the target image and the other images respectively to obtain feature extraction results; The feature extraction result, the target image, the other images and the target identification are stored in the database accordingly.

9. A patrol control device, characterized in that: include: An acquisition module is used to perform multiple image acquisition and image analysis on the same relatively moving target based on the pan-tilt camera carried by the inspection vehicle to obtain multiple detection results, wherein the relatively moving target is used to indicate a target that is in relative motion with the inspection vehicle; A light flow analysis module, used to perform light flow analysis on the multiple detection results by using an optical flow tracking method, to obtain the movement speed and position of the same relatively moving target in a field of view, wherein the field of view is a preset acquisition field of view of the pan-tilt camera; A first angle determination module is used to determine the total rotation angle of the inspection vehicle and the pan-tilt camera based on the movement speed and the position in the field of view, wherein the total rotation angle is the total angle that the inspection vehicle and the pan-tilt camera need to rotate when keeping the same relatively moving target at the center position of the field of view; A second angle determination module is used to determine the rotation angles of the inspection vehicle and the PTZ camera respectively based on the total rotation angle and the speed ratio of the two, wherein the speed ratio of the two is the speed ratio between the inspection vehicle and the PTZ camera; The control module is used to control the rotation of the inspection vehicle and the pan-tilt camera based on the rotation angles corresponding to the inspection vehicle and the pan-tilt camera respectively.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the inspection control method described in any one of claims 1 to 8.

Images