Inspection robot instrument identification method adaptive to light source adjustment

Abstract

According to the inspection robot instrument identification method adaptive to light source adjustment provided by the invention, the illumination condition of the target object is judged and detected by intelligently analyzing the content of the shot image, so that the reading of the instrument is ensured to be clear and visible. Whether the problem of excessive darkness or light reflection exists or not is judged by recognizing the brightness level of the image. Once these conditions are found, the light source is automatically triggered or the position of the light source is adjusted to ensure the clearness of the instrument area. The method solves the problem that the instrument cannot be identified due to external factors in the current instrument identification, can adaptively adjust the light source to reach the instrument identification standard without manual intervention, not only increases the identification accuracy, but also can replace a high-price light sensing camera through the combination of an algorithm and standard binocular camera equipment, and improves the recognition efficiency. The robot cost is effectively reduced, and the automatic inspection robot is more economical and efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision technology, and in particular to an inspection robot instrument recognition method with adaptive light source adjustment. Background Art

[0002] In the context of the intelligent era, the use of computer vision technology has become a key trend in reducing labor costs and improving operational efficiency. Image recognition technology has shown broad application potential in multiple life scenarios. Exploring how to integrate computer vision-driven image recognition technology into patrol robot systems to achieve automated and intelligent efficient patrol inspections is becoming a cutting-edge research topic. An important direction of current research is how to accurately locate instruments in images and analyze their status, such as reading instrument values ​​and judging switch status. Patrol robots equipped with instrument image recognition algorithms have played an important role in unmanned patrol and safety monitoring tasks in places such as substations and data centers. However, the same point instrument will produce dark light and reflection phenomena over time, light, and other factors, which is a technical challenge to be solved at present. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention aims to provide an inspection robot instrument recognition method with adaptive light source adjustment.

[0004] In order to achieve the above object, the present invention adopts the following technical solution: An inspection robot instrument recognition method with adaptive light source adjustment is characterized by comprising the following steps: S1, model training step, collecting historical instrument images taken by the inspection robot to form a data set, and annotating the data set, the annotation information is reflected light, dark light and normal three categories, the annotated data set is divided into a training set, a validation set and a test set, and the divided training set, validation set and test set are used to train the neural network model; S2, image acquisition and recognition step, acquiring the inspection image taken by the inspection robot in real time, cropping it to obtain an instrument image, and using the neural network model obtained in step S1 to recognize the instrument image to confirm its image category; S3, performing a preset process according to the category of the instrument image, and then identifying the content in the instrument image, which specifically includes: S31, when the recognized image category is reflection, first confirm the light source area in the instrument image, then process the light source to move out of the instrument image area, and then recognize the content of the instrument image; S32, when the recognized image category is dark light, turn on the light source and then perform image category recognition. If the recognition result is reflective light, proceed to step S31; if the recognition result is normal, proceed to step S33; S33: If the recognized image category is normal, the content of the instrument image is directly recognized.

[0005] The step S31 of confirming the light source area in the instrument image specifically includes: S311, converting the input instrument image into a grayscale image; S312, binarizing the grayscale image obtained in step S311 by threshold processing, and dividing the image into a reflective area and a non-reflective area, wherein pixels above the threshold are regarded as potential reflective areas; S313, performing an erosion operation on the image binarized in step S312; S314, using a contour detection algorithm to determine the contour of the reflective area in the image, and depicting the contour on the original instrument image; S315, using a deep learning model to obtain size information of the cropped instrument image, and based on the size information, calculate the center point of the instrument image; S316, draw two diagonal lines and a horizontal line and a vertical line passing through the center point on the instrument image to evenly divide the instrument image into eight parts; calculate the area of ​​the reflective region on each part, and set an area threshold. When the area of ​​the reflective region on a part is greater than the area threshold, it is considered that there is a light source area on the part.

[0006] Furthermore, the processing of moving the light source out of the instrument image area in step S31 further includes: S311', when the light source area is only on one part, the direction of the right-angled side a of the part passing through the center point of the instrument image is taken as the moving direction. According to whether the right-angled side a is above or below the center point, the optimal moving route initially obtained is to move the light source upward or downward, and then calculate the distance between the other right-angled side b of the part and the lower boundary of the original inspection image taken by the inspection robot. If the distance is above the set value, the light source is moved according to the obtained optimal moving route, and then the shooting and judgment are continued until it is judged that the light intensity of the part is less than the threshold; if the distance between the right-angled side b and the lower boundary of the original inspection image taken by the inspection robot is less than the set value, the light source is moved in the opposite direction of the optimal moving route, and then the shooting and judgment are continued; if the dial leaves the shooting range after the light source moves, the light source is moved to the original position to continue shooting and judging. If the result of judging and moving the light source for three consecutive times is that the dial is out of the shooting range, the instrument is marked as requiring manual adjustment; S312', when the light source area is on two of the parts, find the overlapping edges of the two parts through the expression of the side length of each part, specifically including: S3121', if the right-angled sides overlap, and the overlapping sides are above or below the center point of the instrument image, the optimal moving route initially obtained is to move the light source upward or downward, and further calculate the distance between the non-overlapping right-angled sides a of the two parts and the lower boundary of the original inspection image taken by the inspection robot. If the distance is above the set value, the light source is moved according to the obtained optimal moving route, and continues to shoot and judge until the light intensity of the part is less than the threshold; if the distance between the non-overlapping right-angled sides a of the two parts and the lower boundary of the original inspection image taken by the inspection robot is less than the set value, the light source is moved in the opposite direction of the optimal moving route and continues to shoot and judge; if the dial leaves the shooting range after moving the light source, the light source is returned to the original position and continues to judge. If the result of moving the light source after five consecutive shots and judgments is that the dial is out of the shooting range, it is marked that the point needs to be manually adjusted; S3122', if the hypotenuse overlaps, when judging the direction of the overlapping edge relative to the center point of the instrument image, the optimal moving route initially obtained is to move the light source in the direction of the overlapping edge relative to the center point of the instrument image, further calculate the angle between the overlapping edge and the x-axis according to the hypotenuse formula, calculate the farthest point of the overlapping edge from the center point of the instrument image, extend the overlapping edge and the original inspection image taken by the inspection robot to intersect at points a and b, if the distance between this point and point b is greater than the set value, move the light source in the direction of the overlapping edge relative to the center point of the instrument image according to the optimal moving route, continue shooting and judging until it is judged that the light intensity of all areas is less than the threshold; if the distance between this point and point b is less than the set value, move the light source downward and continue shooting and judging; if the dial leaves the shooting range after moving the light source, return the light source to the original position and continue shooting and judging. If the result of moving the light source after five consecutive shots and judgments is that the dial leaves the shooting range, mark the point position as requiring manual adjustment; S313', when three parts are detected to obtain light source areas at the same time, the directions of the three parts are determined by the horizontal line or vertical line through the center point of the instrument image. If the three parts are all above the center point of the instrument image, it is preliminarily concluded that the optimal moving route is to move the light source upward; further calculate the distance between the upper boundary of the instrument image and the lower boundary of the original inspection image taken by the inspection robot. If the distance is greater than the set value, move the camera upward according to the optimal moving route, continue shooting and judging, until it is determined that the light intensity of all areas is less than the threshold; if the distance between the upper boundary of the instrument image and the lower boundary of the original inspection image taken by the inspection robot is less than the set value, move the camera downward, and continue shooting and judging. If the dial leaves the shooting range at this time, return the camera to the original position to continue shooting and judging. If the result of moving the camera after five consecutive shots and judgments is that the dial is out of the shooting range, mark the point as requiring manual adjustment; S314', when four parts are detected at the same time to obtain the light source area, the center point of the instrument image is used as the basis for judgment; S3141', through contour analysis, it is determined that the figure surrounded by the four parts is a right triangle, and then it is determined that the image surrounded by the four parts is a right triangle, and the optimal movement route initially obtained is to move the light source to the upper left; the midline of the right triangle is calculated according to the hypotenuse formula, and the midline is extended to intersect with the original inspection image taken by the inspection robot at points a and b. If the distance between this point and point b is greater than the set value, the camera is moved to the upper left according to the optimal movement route, and then the shooting and judgment are continued until it is determined that the light intensity of all areas is less than the threshold value; if the distance is less than the set value, the camera is moved downward to continue the judgment; S3142', if the figure surrounded by the four parts is judged to be a rectangle through contour analysis, then the image surrounded by the four parts is judged to be a rectangle, and on the left side of the instrument image, then the optimal moving route initially obtained is to move the light source to the right, calculate the distance between the right boundary of the instrument image and the left boundary of the original inspection image taken by the inspection robot, if the distance is above the set value, then move the camera to the right according to the optimal moving route, continue shooting and judging, until it is judged that the light intensity of all areas is less than the threshold; if the distance between the right boundary of the instrument image and the left boundary of the original inspection image taken by the inspection robot is less than the set value, then move the camera to the left and continue shooting and judging; if the dial leaves the shooting range after the light source is moved, then return the light source to the original position to continue shooting and judging, if the dial is out of the shooting range after the light source is moved after five consecutive shots and judgments, then mark the point as requiring manual adjustment; S315', if more than five parts have light source areas, then calculate the distances between each side boundary of the instrument image and the corresponding side boundary of the original inspection image taken by the inspection robot, and move the light source in the direction with the smallest distance.

[0007] Furthermore, in step S1, the dataset is annotated by Labelme to generate a corresponding txt file, and the annotated dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0008] Furthermore, in step S3.6, the area of ​​the reflective region on each part is calculated using BoundingRect.

[0009] Another object of the present invention is to propose a patrol robot with adaptive light source adjustment, characterized in that the patrol robot and the light source and camera arranged on the patrol robot move synchronously; a memory for storing computer instructions; and a control module for calling and executing the computer instructions stored in the memory to implement any method in the above technical solutions.

[0010] Another object of the present invention is to provide a computer-readable storage medium, comprising: a memory for storing computer instructions; a controller for calling and executing the computer instructions stored in the memory to implement any method in the above technical solutions.

[0011] The beneficial effects of the present invention are as follows: the present invention intelligently analyzes the captured image content to determine the lighting conditions of the target object to ensure that the meter readings are clear and visible. By identifying the brightness level of the image, it is determined whether there is an excessive dimness or reflection problem. Once these conditions are found, the light source will be automatically triggered or the position of the light source will be adjusted to ensure the clarity of the instrument area. The present invention solves the current situation in which instrument recognition cannot be recognized due to external factors. The light source can be adaptively adjusted to meet the instrument recognition standard without human intervention, which not only increases the recognition accuracy, but also can replace the expensive light-sensing camera through the combination of the algorithm and the standard binocular camera device, effectively reducing the cost of the robot, making the automated inspection robot more economical and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings: Figure 1 A schematic diagram of a flow chart of an inspection robot instrument identification method according to an embodiment of the present invention; Figure 2is a schematic diagram of dividing the instrument image in an embodiment of the present invention; Figure 3 is a schematic diagram of an embodiment of the present invention in which only one part has a light source area; Figure 4 A schematic diagram of an embodiment of the present invention in which two parts have light source areas and the right-angle sides overlap; Figure 5 is a schematic diagram of an embodiment of the present invention in which two parts have light source areas and the hypotenuses overlap; Figure 6 It is a schematic diagram of obtaining a light source area by simultaneously detecting three parts in an embodiment of the present invention; Figure 7-10 In the embodiment of the present invention, there are four parts that simultaneously detect light source areas, and the light source area is a triangle, and the light source area is at different positions of the upper right, lower left, lower right, and upper left; Figure 11-14 These are four situations in which, in an embodiment of the present invention, four parts simultaneously detect light source areas, the light source areas are rectangular, and the light source areas are at different positions, namely, right, left, bottom, and top. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0014] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] The present invention will be further described below in conjunction with the accompanying drawings. It should be noted that this embodiment is based on the technical solution and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to this embodiment.

[0016] like Figure 1 As shown, this embodiment provides an inspection robot instrument recognition method with adaptive light source adjustment, comprising the following steps: S1. Model training step: collect historical instrument images taken by the inspection robot to form a data set, and annotate the data set. The annotation information is three categories: reflective, dark light and normal. The annotated data set is divided into training set, verification set and test set, and the neural network model is trained using the divided training set, verification set and test set.

[0017] In one embodiment, a corresponding txt file can be generated through Labelme annotation, and the annotation information is three categories: reflective, dark light, and normal. The annotated data set is formed into a training set, a verification set, and a test set in a ratio of 8:1:1.

[0018] It should be noted that the recognition and processing of whether an image is reflective, dark or normal by constructing a neural network model can adopt the known technology in the field of images, and the present invention does not limit this. For example, the category of the image can be recognized by preprocessing the input training set, validation set and test set data, extracting features through the convolution layer and pooling layer, integrating features and classifier judgment through the fully connected layer, and optimizing parameters through labeled data training.

[0019] S2, image acquisition and recognition step, acquiring the inspection image taken by the inspection robot in real time, and cropping it to obtain an instrument image, and using the neural network model obtained in step S1 to recognize the instrument image to confirm its image category.

[0020] In one embodiment, the trained neural network model obtained in step S1 is used to identify the inspection image taken by the inspection robot. Preferably, the confidence level is set to 0.8, and only the detection frame with a confidence level >= 0.8 is retained. The detection frame is cropped using its coordinate information to obtain the instrument image.

[0021] S3, performing a preset process according to the category of the instrument image, and then identifying the content in the instrument image, which specifically includes: S31, when the recognized image category is reflection, first confirm the light source area in the instrument image, then process the light source to move out of the instrument image area, and then recognize the content of the instrument image; S32, when the recognized image category is dark light, turn on the light source and then perform image category recognition. If the recognition result is reflective light, proceed to step S31; if the recognition result is normal, proceed to step S33; S33: If the recognized image category is normal, the content of the instrument image is directly recognized.

[0022] It should be noted that how to identify the content of an image is not the technical content of the present invention, and those skilled in the art can implement it through various possible technical methods, which is not limited here.

[0023] In one embodiment, the step S31 of confirming the light source area in the instrument image specifically includes: S311, converting the input instrument image into a grayscale image; S312, binarizing the grayscale image obtained in step S311 by threshold processing, and dividing the image into a reflective area and a non-reflective area, wherein pixels above the threshold are regarded as potential reflective areas; S313, in order to remove noise in the image and clearly identify the reflective area, an erosion operation is performed on the image binarized in step S312. This morphological operation eliminates unnecessary small bright spots through a defined kernel, thereby purifying the image; S314, using a contour detection algorithm to determine the contour of the reflective area in the image, and depicting the contour on the original instrument image, thereby clarifying the boundary of the reflective area; S315, in order to deeply analyze the position and distribution of the reflective area, the size information of the cropped instrument image is obtained using the neural network model obtained in step S1, and the center point of the instrument image is calculated accordingly; S316: Draw two diagonal lines and a horizontal line and a vertical line passing through the center point on the instrument image to evenly divide the instrument image into eight parts, thereby providing a clear spatial reference frame for the reflective area. Figure 2 As shown, 1-8 represent eight equally divided parts. BoundingRect is used to calculate the area of ​​the reflective region on each part, and an area threshold is set. When the area of ​​the reflective region on a part is greater than the area threshold, it is considered that there is a light source area on the part.

[0024] In one embodiment, when the area of ​​the reflective region on a certain portion is greater than 20% of the area of ​​the region, it is considered that there is a light source region on the portion.

[0025] In addition, since the light source and the camera usually move synchronously, it is necessary to move the light source out of the instrument image area through an optimal route.

[0026] In a specific embodiment, the processing of moving the light source out of the instrument image area in step S31 further includes: S311', when the light source area is only on one part, such as Figure 3 When the part "1" is shown, take the shape of part "1" as an example. Figure 3As shown, the expressions of its two right-angled sides a and b are calculated, and the direction of the right-angled side a passing through the center point of the instrument image is the moving direction. The right-angled side a is above the center point, so the optimal moving route initially obtained is to move the light source upward, and calculate the distance between the right-angled side b and the lower boundary of the original inspection image taken by the inspection robot. If the distance is more than 1.5 times the height of the cropped instrument image, then according to the obtained optimal moving route, the light source is moved upward, and then continue to shoot and judge until it is judged that the light intensity of the part "1" is less than the threshold; if the distance between the right-angled side b and the lower boundary of the original inspection image taken by the inspection robot is less than 1.5 times the height of the cropped instrument image, the light source is moved downward, and then continue to shoot and judge; if it is judged that the dial has left the shooting range after moving the light source, the light source is moved to the original position and continues to judge. If the result of three consecutive judgments and moving the camera is that the dial is out of the shooting range, it is marked that the point needs to be manually adjusted.

[0027] S312', when the light source area is on two of the parts, find the overlapping edges of the two parts through the expression of the side length of each part, specifically including: S3121', if the right angles coincide (such as Figure 4 As shown, the light source area is in part "1" and part "2", and the right-angled sides overlap), and the overlapping sides are above the center point of the instrument image. The optimal moving route initially obtained is to move the light source upward, and further calculate the distance between the non-overlapping right-angled side a of the two parts and the lower boundary of the original inspection image taken by the inspection robot. If the distance is more than 1.5 times the height of the instrument image, the light source is moved upward according to the obtained optimal moving route, and the shooting and judgment are continued until the illumination intensity of areas 1 and 2 is less than the threshold; if the distance between the non-overlapping right-angled side a of the two parts and the lower boundary of the original inspection image taken by the inspection robot is less than 1.5 times the height of the instrument image, the light source is moved downward and the judgment is continued. If the dial leaves the shooting range after moving the light source, the light source is moved back to the original position to continue the judgment. If the result of moving the camera after five consecutive shots and judgments is that the dial is out of the shooting range, the point is marked as requiring manual adjustment.

[0028] S3122', if the hypotenuses coincide (such as Figure 5As shown, the light source area is in parts "2" and "3" and the hypotenuse overlaps), when the overlapped edge is in the upper left of the center point of the instrument image, the optimal moving route initially obtained is to move the light source to the upper left, further calculate the angle between the overlapped edge and the x-axis according to the hypotenuse formula, calculate the farthest point of the overlapped edge from the center point of the instrument image, extend the overlapped edge and intersect the original image at points a and b, if the distance between the center point and point b is more than 1.5 times the length of the hypotenuse, move the light source to the upper left according to the optimal moving route, continue shooting and judging, until it is judged that the light intensity of areas 2 and 3 is less than the threshold; if the distance between this point and point b is less than 1.5 times the hypotenuse, move the light source downward and continue shooting and judging. If the dial leaves the shooting range after moving the light source, move the light source back to the original position to continue shooting and judging. If the result of moving the light source after five consecutive shots and judgments is that the dial leaves the shooting range, then mark the point position for manual adjustment.

[0029] S313', when three parts are detected to obtain the light source area at the same time (such as Figure 6 As shown, parts "1", "2" and "3" all have light source areas), then the horizontal line or vertical line through the center point of the instrument image is used to determine the direction of the three parts. If the three parts are all above the center point of the instrument image, it is preliminarily concluded that the optimal moving route is to move the light source upward; further calculate the distance between the upper boundary of the instrument image and the lower boundary of the original inspection image taken by the inspection robot. If the distance is more than 1.5 times the height of the instrument image, the light source is moved upward according to the optimal moving route, and the shooting and judgment are continued until the light intensity of all areas is judged to be less than the threshold; if the distance between the upper boundary of the instrument image and the lower boundary of the original inspection image taken by the inspection robot is less than 1.5 times the height of the instrument image, the light source is moved downward, and the shooting and judgment are continued. If the dial leaves the shooting range after moving the light source, the light source is returned to the original position to continue shooting and judging. If the result of moving the light source after five consecutive shots and judgments is that the dial is out of the shooting range, it is marked that the point needs to be manually adjusted.

[0030] S314', when four parts are detected at the same time to obtain the light source area, the center point of the instrument image is used as the basis for judgment, specifically, including: S3141', through contour analysis, it is determined that the figure enclosed by the four parts is a right triangle, and the optimal movement route initially obtained is to move the light source to the upper left; the midline of the enclosed right triangle is calculated according to the hypotenuse formula, and the midline is extended to intersect with the original inspection image taken by the inspection robot at points a and b. If the distance between this point and point b is above the set value, the camera is moved to the upper left according to the optimal movement route, and then shooting and judging are continued until it is determined that the light intensity of all areas is less than the threshold; if the distance is less than the set value, the camera is moved downward to continue judging.

[0031] exist Figure 7 In the illustrated embodiment, parts "1", "2", "3" and "4" are all areas with light sources. At this time, the image enclosed by the four parts is judged to be a right triangle through contour analysis. The midline of the enclosed right triangle is calculated according to the hypotenuse formula, and the midline is extended to intersect with the original inspection image taken by the inspection robot at points a and b. If the distance between the center point of the disk and the farthest point b is more than 1.5 times the length of the midline of the triangle, the light source is moved to the upper left according to the optimal moving route, and then shooting and judging are continued until it is judged that the light intensity of all areas is less than the threshold; if the distance is less than 1.5 times the length of the midline of the triangle, the light source is moved to the farthest side of the original image with the dial as the target to continue judging.

[0032] exist Figure 8 In the illustrated embodiment, parts "5", "6", "7" and "8" are all areas with light sources. At this time, the image enclosed by the four parts is judged to be a right triangle through contour analysis. The midline of the enclosed right triangle is calculated according to the hypotenuse formula, and the midline is extended to intersect with the original inspection image taken by the inspection robot at points a and b. If the distance between the center point of the disk and the farthest point b is more than 1.5 times the length of the midline of the triangle, the light source is moved to the upper left according to the optimal moving route, and then shooting and judging are continued until it is judged that the light intensity of all areas is less than the threshold; if the distance is less than 1.5 times the length of the midline of the triangle, the light source is moved to the farthest side of the original image with the dial as the target to continue judging.

[0033] exist Fig. 9In the illustrated embodiment, parts "3", "4", "5" and "6" are all areas with light sources. Through contour analysis, it is determined that the image enclosed by the four parts is a right triangle. The midline of the enclosed right triangle is calculated according to the hypotenuse formula, and the midline is extended to intersect with the original inspection image taken by the inspection robot at points a and b. If the distance between the center point of the disk and the farthest point b is more than 1.5 times the length of the midline of the triangle, the light source is moved to the upper left according to the optimal moving route, and then shooting and judging are continued until it is determined that the light intensity of all areas is less than the threshold; if the distance is less than 1.5 times the length of the midline of the triangle, the light source is moved to the farthest side of the original image with the dial as the target to continue judging, as shown by the arrow in the figure.

[0034] exist Fig.10 In the illustrated embodiment, parts "1", "2", "7" and "8" are all areas with light sources. Through contour analysis, it is determined that the image enclosed by the four parts is a right triangle. The midline of the enclosed right triangle is calculated according to the hypotenuse formula, and the midline is extended to intersect with the original inspection image taken by the inspection robot at points a and b. If the distance between the center point of the disk and the farthest point b is more than 1.5 times the length of the midline of the triangle, the light source is moved to the upper left according to the optimal moving route, and then shooting and judging are continued until it is determined that the light intensity of all areas is less than the threshold; if the distance is less than 1.5 times the length of the midline of the triangle, the light source is moved to the farthest side of the original image with the dial as the target to continue judging, as shown by the arrow in the figure.

[0035] S3142', if the figure surrounded by the four parts is judged to be a rectangle through contour analysis, then the image surrounded by the four parts is judged to be a rectangle, and on the left side of the instrument image, then the optimal moving route initially obtained is to move the light source to the right, calculate the distance between the right boundary of the instrument image and the left boundary of the original inspection image taken by the inspection robot, if the distance is above the set value, then move the camera to the right according to the optimal moving route, continue shooting and judging, until it is judged that the light intensity of all areas is less than the threshold; if the distance between the right boundary of the instrument image and the left boundary of the original inspection image taken by the inspection robot is less than the set value, then move the camera to the left and continue shooting and judging; if the dial leaves the shooting range after moving the light source, then return the light source to its original position to continue shooting and judging, if the dial is out of the shooting range after the light source is moved after five consecutive shots and judgments, then mark the point as requiring manual adjustment.

[0036] exist Fig.11In the embodiment shown, parts "2", "3", "4" and "5" are all light source areas. The image surrounded by the four parts is judged to be a rectangle through contour analysis. The median line of the rectangle is calculated and extended to intersect points a and b of the original inspection image taken by the inspection robot. The farthest point is calculated based on the center point of the instrument panel and points a and b. In this figure, it is point b. Then the light source is moved toward point a, and shooting and judgment are continued until it is judged that the light intensity of all areas is less than the threshold. If the center point of the instrument panel is equal to the distance between points a and b, it is moved to the right first. If the dial leaves the shooting range, the light source is returned to the original position to continue shooting and judging. If the dial is out of the shooting range after the light source moves after five consecutive shots and judgments, it is marked that the point needs to be manually adjusted.

[0037] exist Fig.12 In the embodiment shown, parts "1", "6", "7" and "8" are all light source areas. The image surrounded by the four parts is judged to be a rectangle through contour analysis. The median line of the rectangle is calculated and extended to intersect points a and b of the original inspection image taken by the inspection robot. The farthest point is calculated based on the center point of the instrument panel and points a and b. In this figure, it is point b. Then the light source is moved toward point a, and shooting and judgment are continued until it is judged that the light intensity of all areas is less than the threshold. If the center point of the instrument panel is equal to the distance between points a and b, it is moved to the right first. If the dial leaves the shooting range, the light source is returned to the original position to continue shooting and judging. If the dial is out of the shooting range after the light source moves after five consecutive shots and judgments, it is marked that the point needs to be manually adjusted.

[0038] exist Fig.13 In the embodiment shown, parts "4", "5", "6" and "7" are light source areas. The image surrounded by the four parts is judged to be a rectangle through contour analysis. The median line of the rectangle is calculated and extended to intersect points a and b of the original inspection image taken by the inspection robot. The farthest point is calculated based on the center point of the instrument panel and points a and b. In this figure, it is point b. Then the light source is moved toward point a, and shooting and judgment are continued until it is judged that the illumination intensity of all areas is less than the threshold value; if the center point of the instrument panel is equal to the distance between points a and b, it is moved downward first. If the dial leaves the shooting range, the light source is returned to the original position to continue shooting and judging. If the dial is out of the shooting range after the light source moves after five consecutive shots and judgments, it is marked that the point needs to be manually adjusted.

[0039] exist Fig.14In the embodiment shown, parts "1", "2", "3" and "8" are light source areas. The image surrounded by the four parts is judged to be a rectangle through contour analysis. The median line of the rectangle is calculated and extended to intersect points a and b of the original inspection image taken by the inspection robot. The farthest point is calculated based on the center point of the instrument panel and points a and b. In this figure, it is point b. Then the light source is moved toward point a, and shooting and judgment are continued until it is judged that the illumination intensity of all areas is less than the threshold value; if the center point of the instrument panel is equal to the distance between points a and b, it is moved downward first. If the dial leaves the shooting range, the light source is returned to the original position to continue shooting and judging. If the dial is out of the shooting range after the light source moves after five consecutive shots and judgments, it is marked that the point needs to be manually adjusted.

[0040] S315', if more than five parts have light source areas, then calculate the distances between each side boundary of the instrument image and the corresponding side boundary of the original inspection image taken by the inspection robot, and move the light source in the direction with the smallest distance.

[0041] In addition, in another embodiment of the present disclosure, a patrol robot with adaptive light source adjustment is proposed, which is characterized by the patrol robot and the light source and camera arranged on the patrol robot, and the light source and the camera move synchronously; a memory for storing computer instructions; and a control module for calling and executing the computer instructions stored in the memory to implement any method in the above technical solutions.

[0042] In another embodiment of the present disclosure, a computer-readable storage medium is proposed, including: a memory for storing computer instructions; a controller for calling and executing the computer instructions stored in the memory to implement any method in the above technical solutions.

[0043] For those skilled in the art, various corresponding changes and modifications can be made according to the above technical solutions and concepts, and all of these changes and modifications should be included in the protection scope of the claims of the present invention.

Claims

1. A patrol robot instrument recognition method with adaptive light source adjustment, characterized in that: The following steps are involved: S1, model training step, collecting historical instrument images taken by the inspection robot to form a data set, and annotating the data set, the annotation information is reflected light, dark light and normal three categories, the annotated data set is divided into a training set, a validation set and a test set, and the divided training set, validation set and test set are used to train the neural network model; S2, image acquisition and recognition step, acquiring the inspection image taken by the inspection robot in real time, cropping it to obtain an instrument image, and using the neural network model obtained in step S1 to recognize the instrument image to confirm its image category; S3, performing a preset process according to the category of the instrument image, and then identifying the content in the instrument image, which specifically includes: S31, when the recognized image category is reflection, first confirm the light source area in the instrument image, then process the light source to move out of the instrument image area, and then recognize the content of the instrument image; S32, when the recognized image category is dark light, turn on the light source and then perform image category recognition. If the recognition result is reflective light, proceed to step S31; if the recognition result is normal, proceed to step S33; S33, if the recognized image category is normal, directly recognize the content of the instrument image; The step S31 of confirming the light source area in the instrument image specifically includes: S311, converting the input instrument image into a grayscale image; S312, binarizing the grayscale image obtained in step S311 by threshold processing, and dividing the image into a reflective area and a non-reflective area, wherein pixels above the threshold are regarded as potential reflective areas; S313, performing an erosion operation on the image binarized in step S312; S314, using a contour detection algorithm to determine the contour of the reflective area in the image, and depicting the contour on the original instrument image; S315, using a deep learning model to obtain size information of the cropped instrument image, and based on the size information, calculate the center point of the instrument image; S316. Draw two diagonal lines and a horizontal line and a vertical line passing through a center point on the instrument image to evenly divide the instrument image into eight parts; calculate the area of ​​the reflective region on each part, and set an area threshold. When the area of ​​the reflective region on a part is greater than the area threshold, it is considered that there is a light source area on the part.

2. The method according to claim 1, characterized in that The processing of moving the light source out of the instrument image area in step S31 includes: S311', when the light source area is only on one part, the direction of the right-angled side a of the part passing through the center point of the instrument image is taken as the moving direction. According to whether the right-angled side a is above or below the center point, the optimal moving route initially obtained is to move the light source upward or downward, and then calculate the distance between the other right-angled side b of the part and the lower boundary of the original inspection image taken by the inspection robot. If the distance is above the set value, the light source is moved according to the obtained optimal moving route, and then the shooting and judgment are continued until it is judged that the light intensity of the part is less than the threshold; if the distance between the right-angled side b and the lower boundary of the original inspection image taken by the inspection robot is less than the set value, the light source is moved in the opposite direction of the optimal moving route, and then the shooting and judgment are continued; if the dial leaves the shooting range after the light source moves, the light source is moved to the original position to continue shooting and judging. If the result of judging and moving the light source for three consecutive times is that the dial is out of the shooting range, the instrument is marked as requiring manual adjustment; S312', when the light source area is on two of the parts, find the overlapping edges of the two parts through the expression of the side length of each part, specifically including: S3121', if the right-angled sides overlap, and the overlapping sides are above or below the center point of the instrument image, the optimal moving route initially obtained is to move the light source upward or downward, and further calculate the distance between the non-overlapping right-angled sides a of the two parts and the lower boundary of the original inspection image taken by the inspection robot. If the distance is above the set value, the light source is moved according to the obtained optimal moving route, and continues to shoot and judge until the illumination intensity of the two parts is less than the threshold; if the distance between the non-overlapping right-angled sides a of the two parts and the lower boundary of the original inspection image taken by the inspection robot is less than the set value, the light source is moved in the opposite direction of the optimal moving route and continues to shoot and judge; if the dial leaves the shooting range after moving the light source, the light source is returned to the original position and continues to judge. If the result of moving the light source after five consecutive shots and judgments is that the dial is out of the shooting range, it is marked that the point needs to be manually adjusted; S3122', if the hypotenuse overlaps, when judging the direction of the overlapping edge relative to the center point of the instrument image, the optimal moving route initially obtained is to move the light source in the direction of the overlapping edge relative to the center point of the instrument image, further calculate the angle between the overlapping edge and the x-axis according to the hypotenuse formula, calculate the farthest point of the overlapping edge from the center point of the instrument image, extend the overlapping edge and the original inspection image taken by the inspection robot to intersect at points a and b, if the distance between this point and point b is greater than the set value, move the light source in the direction of the overlapping edge relative to the center point of the instrument image according to the optimal moving route, continue shooting and judging until it is judged that the illumination intensity of the two parts is less than the threshold; if the distance between this point and point b is less than the set value, move the light source downward and continue shooting and judging; if the dial leaves the shooting range after moving the light source, return the light source to the original position and continue shooting and judging. If the result of moving the light source after five consecutive shots and judgments is that the dial leaves the shooting range, mark the point position as requiring manual adjustment; S313', when three parts are detected to obtain light source areas at the same time, the directions of the three parts are determined by the horizontal line or vertical line through the center point of the instrument image. If the three parts are all above the center point of the instrument image, it is preliminarily concluded that the optimal moving route is to move the light source upward; further calculate the distance between the upper boundary of the instrument image and the lower boundary of the original inspection image taken by the inspection robot. If the distance is greater than the set value, move the camera upward according to the optimal moving route, continue shooting and judging, until it is determined that the light intensity of all areas is less than the threshold; if the distance between the upper boundary of the instrument image and the lower boundary of the original inspection image taken by the inspection robot is less than the set value, move the camera downward, and continue shooting and judging. If the dial leaves the shooting range at this time, return the camera to the original position to continue shooting and judging. If the result of moving the camera after five consecutive shots and judgments is that the dial is out of the shooting range, mark the point as requiring manual adjustment; S314', when four parts are detected at the same time to obtain the light source area, the center point of the instrument image is used as the basis for judgment; S3141', through contour analysis, it is determined that the figure surrounded by the four parts is a right triangle, and the optimal movement route initially obtained is to move the light source to the upper left; the midline of the right triangle is calculated according to the hypotenuse formula, and the midline is extended to intersect with the original inspection image taken by the inspection robot at points a and b. If the distance between this point and point b is greater than the set value, the camera is moved to the upper left according to the optimal movement route, and then continues to shoot and judge until it is determined that the light intensity of all areas is less than the threshold; if the distance is less than the set value, the camera is moved downward to continue judging; S3142', if the figure surrounded by the four parts is judged to be a rectangle through contour analysis, then the image surrounded by the four parts is judged to be a rectangle, and on the left side of the instrument image, then the optimal moving route initially obtained is to move the light source to the right, calculate the distance between the right boundary of the instrument image and the left boundary of the original inspection image taken by the inspection robot, if the distance is above the set value, then move the camera to the right according to the optimal moving route, continue shooting and judging, until it is judged that the light intensity of all areas is less than the threshold; if the distance between the right boundary of the instrument image and the left boundary of the original inspection image taken by the inspection robot is less than the set value, then move the camera to the left and continue shooting and judging; if the dial leaves the shooting range after the light source is moved, then return the light source to the original position to continue shooting and judging, if the dial is out of the shooting range after the light source is moved after five consecutive shots and judgments, then mark the point as requiring manual adjustment; S315', if more than five parts have light source areas, then calculate the distances between each side boundary of the instrument image and the corresponding side boundary of the original inspection image taken by the inspection robot, and move the light source in the direction with the smallest distance.

3. The method according to claim 2, characterized in that In step S1, Labelme is used to annotate the data set to generate the corresponding txt file, and the annotated data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

4. The method according to claim 3, characterized in that In step S316, the area of ​​the reflective region on each part is calculated using BoundingRect.

5. An inspection robot with adaptive light source adjustment, characterized in that: An inspection robot and a light source and a camera arranged on the inspection robot, the light source and the camera moving synchronously; a memory for storing computer instructions; a control module for calling and executing the computer instructions stored in the memory to implement the method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that: include: Memory, for storing computer instructions; A controller, configured to call and execute computer instructions stored in the memory to implement the method according to any one of claims 1 to 4.

Images