Robot video stream inspection method based on AI real-time vision

Abstract

The invention discloses a robot video stream inspection method based on AI real-time vision, and the method comprises the steps: 1, obtaining an inspection route, and dividing the inspection route into a plurality of non-stop route segments according to different holder angles of a robot; 2, the robot starts to inspect along the inspection route, when the robot is located at the identification starting point, the robot configures motion parameters of the robot, parameters of the pan-tilt camera and a target needing to be identified in the route section, then the robot moves along the inspection route, meanwhile, the pan-tilt camera starts to acquire target image information until the robot moves to the identification ending point, and identification is completed; the pan-tilt camera stops working; and step 3, identifying the target image information acquired by the pan-tilt camera, and judging whether there is an abnormality or not. According to the scheme, the visible light imaging definition in the movement process of the robot is guaranteed, it is guaranteed that the robot can still keep a clear shot picture in the continuous high-speed movement state, and the accuracy and reliability of image recognition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of inspection robots, and in particular to a robot video stream inspection method based on AI real-time vision. Background Art

[0002] At present, intelligent inspection robots usually work by taking photos and identifying at fixed points. Normal inspection tasks that require taking pictures / video segments at fixed points for identification have the following limitations when using this type of inspection method for image algorithm recognition: 1. Low efficiency: When the current inspection robot performs a single fixed-point image recognition task, it involves a series of processes such as pan-tilt rotation operation, fine adjustment of camera parameters, and screenshots. When facing multiple detection points, the cumulative single full-path inspection cycle may exceed the 1-hour threshold, which makes it unable to fully meet the needs of efficient inspection operations in specific application scenarios, such as the protection of robot components in high-temperature scenarios. Taking a specific project as an example, the robot patrol speed is 0.4m / s, the average time required for single-point fixed-point shooting is about 15 seconds, and the total length of the inspection route is 120 meters. Since the points in the area to be inspected in this project are relatively dense, a conventional inspection method is adopted, and a stop point needs to be set every 1 meter. Each stop point is calculated according to 2 devices. It is concluded that the fixed-point shooting process alone takes about 60 minutes, the road section inspection takes 5 minutes, and other scheduling logic processing takes 10 minutes. The total inspection time is relatively long, and the overall inspection efficiency is low.

[0003] 2. Low flexibility: The current system requires that all stop locations be pre-set and accurately calibrated. For each location change or new identification point, on-site staff must re-perform the tedious point calibration process. This working mechanism limits the system's ability to respond quickly and flexibly to changing and complex situations in usage scenarios. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects in the prior art that the inspection robot needs to stop at the inspection point for inspection, the total inspection time is relatively long, the overall inspection efficiency is low, and the requirement that the inspection time in a specific environment cannot exceed the time limit is not met. A robot video stream inspection method based on AI real-time vision is provided.

[0005] The purpose of the present invention is achieved through the following technical solutions: A robot video stream inspection method based on AI real-time vision includes the following steps: Step 1: Obtain the inspection route, divide the inspection route into several non-stop route segments according to the different pan / tilt angles of the robot, and preset the start and end recognition points for each route segment; Step 2: The robot starts to inspect along the inspection route. When it is at the start recognition point, the robot configures its own motion parameters, the parameters of the PTZ camera, and the target to be recognized in the current route segment. Then the robot moves along the inspection route, and the PTZ camera starts to acquire target image information. The robot stops working when it moves to the end recognition point. Step 3, identify the target image information obtained by the PTZ camera to determine whether there is any abnormality. If there is an abnormality, send it to the remote end for reminder. If there is no abnormality, continue to inspect along the inspection route until the inspection task is completed.

[0006] Preferably, in the step 2, configuring the parameters of the gimbal camera specifically includes adjusting the relationship between the magnification and eccentricity of the gimbal camera through a magnification-eccentricity model, adjusting the conversion relationship between the gimbal motion and the pixel coordinates of the gimbal camera through a gimbal-pixel conversion model, and configuring the relationship parameters of the robot. Preferably, the relationship parameters of the robot include movement speed, pan / tilt camera focal length, magnification and object distance.

[0007] Preferably, the magnification-eccentricity model is specifically: Among them, P zn For different magnifications, P z1 is the base magnification, is the total eccentricity, It represents the change factor of the eccentricity from the n-1th magnification to the nth magnification, and so on. Preferably, the pan / tilt-pixel conversion model is specifically: A1=[t x t y ]×Δ1+A cur , Among them, [t x t y ] is the offset of the target frame from the image center, A cur The current gimbal angle.

[0008] Preferably, in the step 3, the target image information acquired by the gimbal camera is identified, specifically: a positioning frame extraction scheme is constructed to obtain the positioning information of the robot at preset time intervals, and based on the positioning sampling frequency and running speed, a timer is designed to predict the robot's travel distance to realize real-time image information positioning frame extraction recognition.

[0009] Preferably, in the step 3, the target image information acquired by the gimbal camera is identified on the background server, and the robot transmits the target image information to the background server via a network communication connection; the robot detects the network connection status, and if a network interruption is detected, the robot stores the acquired target image information locally, and when the network is restored, the robot automatically uploads the accumulated local target image information data to the background server.

[0010] Preferably, the robot performs different types of image recognition in different non-stop route segments through the pipeline configuration management function.

[0011] As a preferred embodiment, the robot video stream inspection method based on AI real-time vision also optimizes the method for identifying the target image information obtained by the pan-tilt camera in step 3, specifically: Step 3a, obtaining all target image information from the start recognition point to the end recognition point, and performing preliminary recognition on the target image information; Step 3b, determining the number of images with abnormalities in the preliminary recognition of all target image information, if the proportion of images with abnormalities is greater than or equal to the set threshold, jump to step 3c, if the proportion of images with abnormalities is less than the set threshold, jump to step 3d; Step 3c, extracting abnormal feature points from the abnormal image, comparing the abnormal feature points with the pre-stored feature points of the noise item, if the similarity between the abnormal feature points and the feature points of the noise item is less than a set value, then the target image information is judged to be abnormal, otherwise it is judged to be interference from the noise item and the target image information is normal; Step 3d, determine whether the abnormal image is continuous. If the image is continuous, compare the abnormal feature points with the feature points of the pre-stored noise items. If the similarity between the abnormal feature points and the feature points of the noise items is less than the set value, the target image information is judged to be abnormal. Otherwise, it is judged to be interference from the noise item and the target image information is normal. If the image is discontinuous, the target image information is judged to be normal.

[0012] The beneficial effects of the present invention are as follows: the solution of the present invention enables the robot to continuously collect image data during movement, avoiding recognition gaps when switching between different preset points, ensuring seamless coverage of the entire monitored section, and thus greatly improving recognition efficiency in space and time; at the same time, the present invention does not require manual setting of multiple fixed shooting positions, reducing the time cost of personnel deployment and inspection, and improving overall operation and maintenance efficiency. Furthermore, the solution of the present invention ensures the clarity of visible light imaging during the robot's movement, ensuring that the robot can still maintain a clear shooting image when in continuous high-speed motion, and improving the accuracy and reliability of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the present invention; Figure 2 It is a schematic diagram of an implementation method of video stream inspection of the present invention. DETAILED DESCRIPTION

[0014] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0015] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0016] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0017] Embodiment 1: A robot video stream inspection method based on AI real-time vision, such as Figure 1 As shown, the following steps are included: Step 1: Get the inspection route, and divide the inspection route into several non-stop route segments according to the different pan / tilt angles of the robot. Each route segment has a preset start and end recognition point, such as Figure 2 As shown; Step 2: The robot starts to inspect along the inspection route. When it is at the start recognition point, the robot configures its own motion parameters, the parameters of the PTZ camera, and the target to be recognized in the current route segment. Then the robot moves along the inspection route, and the PTZ camera starts to acquire target image information. The robot stops working when it moves to the end recognition point. Step 3, identify the target image information obtained by the PTZ camera to determine whether there is any abnormality. If there is an abnormality, send it to the remote end for reminder. If there is no abnormality, continue to inspect along the inspection route until the inspection task is completed.

[0018] In the step 2, configuring the parameters of the gimbal camera specifically includes adjusting the relationship between the magnification and eccentricity of the gimbal camera through a magnification-eccentricity model, adjusting the conversion relationship between the gimbal motion and the pixel coordinates of the gimbal camera through a gimbal-pixel conversion model, and configuring the relationship parameters between the robot motion speed, the focal length, magnification, and object distance of the gimbal camera.

[0019] In this solution, the robot continuously shoots videos containing target images while in high-speed motion. The video consists of multiple frames, each of which contains target image information. By configuring the parameters of the gimbal camera, the robot can still shoot clear targets while in continuous high-speed motion, ensuring the accuracy and reliability of image recognition. The route segments are divided according to the different gimbal angles of the robot, which means that the inspection routes and identification points can be flexibly adjusted according to the actual environment and needs to adapt to different inspection scenarios and task requirements. Due to the use of AI real-time vision technology, the robot can obtain and analyze image information in real time during the inspection process, promptly discover and handle abnormal situations, and improve the real-time and response speed of the inspection.

[0020] The magnification-eccentricity model is specifically: Among them, P zn For different magnifications, P z1 is the base magnification, is the total eccentricity, It represents the change factor of the eccentricity from the n-1th magnification to the nth magnification, and so on; The pan / tilt-pixel conversion model is specifically: A1=[t x t y ]×Δ1+A cur , Among them, [t x t y ] is the offset of the target frame from the image center, A cur The current gimbal angle.

[0021] In the step 3, the target image information obtained by the PTZ camera is identified, specifically: Construct a positioning frame extraction scheme to obtain the robot's positioning information at preset time intervals. Based on the positioning sampling frequency and running speed, design a timer to predict the robot's travel distance to achieve real-time image information positioning frame extraction recognition.

[0022] In the application scenario, it is necessary to achieve accurate matching between the recognition results and the specific location information. Not only should the target object or abnormal state be accurately identified, but also these recognition results should be marked in real time at the actual location where they occur, so as to quickly locate the problem and provide accurate guidance for subsequent analysis and processing and on-site intervention. In order to solve the problem of matching the target object and the actual location, this solution combines the robot motion design positioning frame extraction scheme, obtains the robot positioning information at 200ms~500ms, and designs a timer based on the positioning sampling frequency and running speed to predict the robot's travel distance to realize real-time video stream positioning frame extraction recognition. According to the characteristics of the target detection object and tracking the target object, the key frames of the video stream are extracted for accurate image recognition, which solves the problem of poor positioning frame extraction caused by dense equipment and network delay in specific scenarios.

[0023] In the step 3, the target image information obtained by the gimbal camera is identified on the background server, and the robot transmits the target image information to the background server through a network communication connection; the robot detects the network connection status, and if a network interruption is detected, the robot stores the obtained target image information locally, and when the network is restored, the robot automatically uploads the accumulated local target image information data to the background server.

[0024] In the process of using the background video stream recognition service to perform algorithm recognition calculations, ensuring the consistency of video stream data transmission with the robot side is a technical challenge. In order to effectively deal with network anomalies, this solution introduces a fault protection strategy. This mechanism can continuously monitor the network connection status and immediately initiate an emergency response once a network interruption is detected. On this basis, this solution designs a local data sampling and temporary storage solution. Even during a network interruption, the robot will not stop working, but will instead collect and temporarily store video data locally. This instant data preservation strategy can ensure that important information will not be lost during periods of no network connection. Once the network environment returns to stability, the system will automatically upload the accumulated local data and seamlessly connect to the background recognition service to continue the recognition task, reducing task interruptions and data omissions caused by network dependence, and enhancing the robustness of the system and the continuity of task execution.

[0025] The robot performs different types of image recognition in different non-stop route segments through pipeline configuration management functions.

[0026] Due to the complexity of the application scenario, it is required that in a single mission, multiple non-stop areas can perform multiple different types of algorithm recognition, that is, support the dynamic binding and loading of the robot camera's recognition tasks according to the different characteristics of these areas. Specifically, the principle of this solution is to unify the abstraction of the common demand parts, and design the pipeline configuration management function with an integrated idea, which supports the completion of the pipeline reasoning process design in the configuration file. It also supports the configuration of multiple pipelines, each pipeline completes an independent function, and the outputs of multiple pipelines can be connected to the same pipeline, or the output of a pipeline can be connected to multiple pipelines. At the same time, it can also dynamically add, modify, connect, and delete pipelines based on communication protocol control. Based on the pipeline configuration management function, the system function combination can be conveniently performed based on the algorithm configuration. For example, in a certain project, area 1 performs crystallization recognition, area 2 performs water leakage recognition, and area 3 performs electrolytic cell temperature recognition.

[0027] The robot video stream inspection method based on AI real-time vision also optimizes the method of identifying the target image information obtained by the pan-tilt camera in step 3, specifically: Step 3a, obtaining all target image information from the start recognition point to the end recognition point, and performing preliminary recognition on the target image information; Step 3b, determining the number of images with abnormalities in the preliminary recognition of all target image information, if the proportion of images with abnormalities is greater than or equal to the set threshold, jump to step 3c, if the proportion of images with abnormalities is less than the set threshold, jump to step 3d; Step 3c, extracting abnormal feature points from the abnormal image, comparing the abnormal feature points with the pre-stored feature points of the noise item, if the similarity between the abnormal feature points and the feature points of the noise item is less than a set value, then the target image information is judged to be abnormal, otherwise it is judged to be interference from the noise item and the target image information is normal; Step 3d, determine whether the abnormal image is continuous. If the image is continuous, compare the abnormal feature points with the feature points of the pre-stored noise items. If the similarity between the abnormal feature points and the feature points of the noise items is less than the set value, the target image information is judged to be abnormal. Otherwise, it is judged to be interference from the noise item and the target image information is normal. If the image is discontinuous, the target image information is judged to be normal.

[0028] In this scheme, the noise items may include environmental noise: light changes (too bright or too dark), shadows, reflections, rain and snow weather, etc., which may cause noise or blur in the target image information obtained by the pan-tilt camera; motion noise: the robot may vibrate or shake during the inspection process, and this change in motion state may cause blur or displacement in the image captured by the pan-tilt camera; there may also be other interference items, such as temporary obstacles on the inspection route, animals passing by, human interference, etc., which may affect the image information obtained by the pan-tilt camera. The design of this scheme takes into account the impact of noise items on target image recognition and eliminates the interference of noise items on image recognition. In order to improve efficiency, this scheme classifies the proportion of images with abnormalities in all target image information. When it is greater than or equal to the set threshold, it can be judged that the abnormal situation basically exists, but it cannot be determined whether it is a noise item or a real abnormality, and then judge whether it is a noise item. In another case, when it is less than the set threshold, first judge whether the image with abnormality is continuous. If it is a discontinuous image, it can be judged as occasional noise interference, and there is no need to make a special judgment through feature points, which improves the efficiency of identifying abnormalities. In addition, the present solution can also learn the noise item. For the noise feature points that are not pre-stored, they can be recorded and learned after the recognition error, so as to ensure better recognition of the noise item in the future image recognition.

[0029] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0030] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A robot video stream inspection method based on AI real-time vision, characterized in that: The following steps are involved: Step 1: Obtain the inspection route, divide the inspection route into several non-stop route segments according to the different pan / tilt angles of the robot, and preset the start and end recognition points for each route segment; Step 2: The robot starts to inspect along the inspection route. When it is at the start recognition point, the robot configures its own motion parameters, the parameters of the PTZ camera, and the target to be recognized in the current route segment. Then the robot moves along the inspection route, and the PTZ camera starts to acquire target image information. The robot stops working when it moves to the end recognition point. Step 3, identify the target image information obtained by the PTZ camera to determine whether there is any abnormality. If there is an abnormality, send it to the remote end for reminder. If there is no abnormality, continue to inspect along the inspection route until the inspection task is completed.

2. According to the AI ​​real-time vision-based robot video stream inspection method of claim 1, it is characterized in that: In the step 2, configuring the parameters of the gimbal camera specifically includes adjusting the relationship between the magnification and eccentricity of the gimbal camera through a magnification-eccentricity model, adjusting the conversion relationship between the gimbal motion and the pixel coordinates of the gimbal camera through a gimbal-pixel conversion model, and configuring the relationship parameters of the robot.

3. The robot video stream inspection method based on AI real-time vision according to claim 2 is characterized in that: The relationship parameters of the robot include movement speed, pan / tilt camera focal length, magnification and object distance.

4. The robot video stream inspection method based on AI real-time vision according to claim 2 is characterized in that: The magnification-eccentricity model is specifically: Among them, Pzn is different magnification, P z1 is the base magnification, is the total eccentricity, It represents the change factor of the eccentricity from the n-1th magnification to the nth magnification, and so on.

5. The robot video stream inspection method based on AI real-time vision according to claim 2 is characterized in that: The pan / tilt-pixel conversion model is specifically: A1=[t x t y ]×Δ1+A cur , Among them, [t x t y ] is the offset of the target frame from the image center, A cur The current gimbal angle.

6. The robot video stream inspection method based on AI real-time vision according to claim 1 is characterized in that: In the step 3, the target image information obtained by the PTZ camera is identified, specifically: Construct a positioning frame extraction scheme to obtain the robot's positioning information at preset time intervals. Based on the positioning sampling frequency and running speed, design a timer to predict the robot's travel distance to achieve real-time image information positioning frame extraction recognition.

7. The robot video stream inspection method based on AI real-time vision according to claim 1 is characterized in that: In the step 3, the target image information obtained by the gimbal camera is identified on the background server, and the robot transmits the target image information to the background server through a network communication connection; the robot detects the network connection status, and if a network interruption is detected, the robot stores the obtained target image information locally, and when the network is restored, the robot automatically uploads the accumulated local target image information data to the background server.

8. The robot video stream inspection method based on AI real-time vision according to claim 1 is characterized in that: The robot performs different types of image recognition in different non-stop route segments through pipeline configuration management functions.

9. The robot video stream inspection method based on AI real-time vision according to claim 8 is characterized in that: The pipeline configuration management function includes connecting the outputs of multiple pipelines to the same pipeline, or connecting the output of one pipeline to multiple pipelines, and dynamically adding, modifying, connecting, and deleting pipelines based on communication protocol control.

10. The robot video stream inspection method based on AI real-time vision according to claim 1 is characterized in that: The method for identifying the target image information obtained by the gimbal camera in step 3 is also optimized, specifically: Step 3a, obtaining all target image information from the start recognition point to the end recognition point, and performing preliminary recognition on the target image information; Step 3b, determining the number of images with abnormalities in the preliminary recognition of all target image information, if the proportion of images with abnormalities is greater than or equal to the set threshold, jump to step 3c, if the proportion of images with abnormalities is less than the set threshold, jump to step 3d; Step 3c, extracting abnormal feature points from the abnormal image, comparing the abnormal feature points with the pre-stored feature points of the noise item, if the similarity between the abnormal feature points and the feature points of the noise item is less than a set value, then the target image information is judged to be abnormal, otherwise it is judged to be interference from the noise item and the target image information is normal; Step 3d, determine whether the abnormal image is continuous. If the image is continuous, compare the abnormal feature points with the feature points of the pre-stored noise items. If the similarity between the abnormal feature points and the feature points of the noise items is less than the set value, the target image information is judged to be abnormal. Otherwise, it is judged to be interference from the noise item and the target image information is normal. If the image is discontinuous, the target image information is judged to be normal.

Images