Robot inspection method and system for dynamic scene

Abstract

The invention relates to the technical field of robots, in particular to a robot inspection method and system for a dynamic scene. The method comprises the following steps: acquiring an infrared image of a target place, and taking an area of which the temperature is greater than a temperature threshold value in the infrared image as a candidate area; collecting a visible light image of the candidate area; inputting the infrared image into a first classifier to obtain a first confidence coefficient; inputting the visible light image into a second classifier or a neural network model to obtain a second confidence coefficient; fusing the first confidence coefficient and the second confidence coefficient based on a D-S evidence theory to obtain a smoking behavior confidence coefficient; responding to the condition that the smoking behavior confidence is greater than a first threshold value, and judging that the smoking behavior exists; and in response to the condition that the smoking behavior confidence is less than or equal to a first threshold value and greater than a second threshold value, performing smoking behavior modeling based on a Markov chain to determine whether a smoking behavior exists or not. The accuracy of smoking detection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology. More specifically, the present invention relates to a robot inspection method and system for dynamic scenes. Background Art

[0002] In public places where smoking is prohibited, such as hospitals, shopping malls, museums, and schools, it is necessary to promptly detect smoking and issue warnings to encourage smokers to stop. Robots can be used to patrol these no-smoking areas. During the inspection, the robots determine whether smoking is occurring in the prohibited areas and, if so, provide timely feedback to the backend to prevent accidents.

[0003] Existing technologies typically use infrared or visible light cameras mounted on patrol robots to detect smoking. However, the infrared thermal images generated by infrared cameras struggle to distinguish between cigarette lights and other small, hot objects, such as lighters, leading to misjudgments. Visible light cameras, on the other hand, rely on lighting conditions and are prone to failure at night or in obscured environments, resulting in misjudgments and missed detections. Furthermore, existing methods often rely on static image analysis, making them inadequate for tracking dynamic targets while the robot is moving. They are also susceptible to environmental interference, such as steam and dust, which can lead to misjudgments.

[0004] In this regard, how to avoid misjudgments and missed judgments caused by single sensors, environmental interference, and static image analysis, so as to improve the accuracy of smoking detection during robot inspections, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In order to solve the technical problem of low accuracy of smoking detection caused by a single sensor, environmental interference and static image analysis, the present invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a robot inspection method for dynamic scenes, comprising: collecting an infrared image of a target location, and selecting an area in the infrared image whose temperature is greater than a temperature threshold as a candidate area; collecting a visible light image of the candidate area; inputting the infrared image into a preset first classifier to obtain a first confidence level; inputting the visible light image into a preset second classifier or a neural network model to obtain a second confidence level; fusing the first confidence level and the second confidence level based on the DS evidence theory to obtain a smoking behavior confidence level; in response to the smoking behavior confidence level being greater than a first threshold, determining that smoking behavior exists and outputting a warning; in response to the smoking behavior confidence level being less than or equal to a first threshold level and greater than a second threshold level, performing smoking behavior modeling based on a Markov chain to further determine whether smoking behavior exists.

[0007] Furthermore, the first confidence level and the second confidence level are integrated based on the DS evidence theory, including: determining the conflict coefficient between the first confidence level and the second confidence level; determining the smoking behavior confidence level based on the conflict coefficient, the first confidence level and the second confidence level, and the smoking behavior confidence level is negatively correlated with the conflict coefficient, the first confidence level and the second confidence level, respectively.

[0008] Furthermore, the calculation expression of the smoking behavior confidence is:

[0009] C=(1-K)C1·C2;

[0010] In the formula, C is the smoking behavior confidence, C1 is the first confidence, C2 is the second confidence, and K is the conflict coefficient.

[0011] Furthermore, smoking behavior modeling is performed based on a Markov chain, including: defining smoking states as lighting, holding a cigarette, exhaling, and ending; obtaining a state transition probability matrix based on historical data training; acquiring multiple frames of continuous visible light images, and determining that smoking behavior exists if the smoking probability in response to the multiple frames of visible light images is greater than a preset threshold; the smoking probability is determined by a first probability and / or a second probability, the first probability representing the probability of transitioning from the lighting state to the holding cigarette state, and the second probability representing the probability of transitioning from the holding cigarette state to the exhaling cigarette state.

[0012] Furthermore, the method for determining the smoking probability includes: multiplying the first probability by the second probability to obtain the smoking probability.

[0013] Furthermore, before performing smoking behavior modeling based on the Markov chain, the method further includes: determining an optimal observation distance according to the smoking behavior confidence, and moving based on the optimal observation distance, wherein the optimal observation distance is negatively correlated with the smoking behavior confidence.

[0014] Furthermore, the calculation expression of the optimal observation distance is:

[0015] d=d min +(d max -d min )(1-C);

[0016] Where d is the optimal observation distance, d min is the preset first distance, d max The second distance is preset, d min <d max , C is the confidence level of the smoking behavior.

[0017] Furthermore, moving based on the optimal observation distance also includes: generating an obstacle avoidance path according to minimizing a cost function; the cost function is obtained by weighted summation of the smoking behavior confidence, the distance to the obstacle, and the angle required to rotate; and moving based on the obstacle avoidance path.

[0018] Furthermore, the method further includes: determining the frequency of inspection of each target location according to the frequency of detecting smoking behavior in each target location.

[0019] In a second aspect, the present invention provides a robot inspection system for dynamic scenes, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a robot inspection method for dynamic scenes according to any one of the first aspects is implemented.

[0020] The beneficial effects of the present invention are as follows: by fusing data from infrared and visible light cameras for smoke detection, the present invention avoids the misjudgment and missed detection that can easily occur with a single sensor during smoke detection; by dynamically modeling smoking behavior, the present invention can adapt to target tracking during robot movement, avoid environmental interference, and improve the accuracy of smoke detection. By adjusting the optimal observation distance of the inspection robot before dynamic modeling, it can avoid misjudgment and missed detection caused by excessive observation distance, thereby improving the accuracy of robot inspections; by collecting visible light images of candidate areas, it can avoid interference with visible light image recognition caused by other features outside the candidate area, thereby improving the accuracy of smoke detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart schematically illustrating a robot inspection method for dynamic scenes according to an embodiment of the present invention;

[0022] Figure 2 FIG. 4 is a block diagram schematically illustrating a structure of a robot inspection system for dynamic scenes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0024] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1FIG. 4 is a flow chart schematically illustrating a robot inspection method for dynamic scenes according to an embodiment of the present invention.

[0026] In the first aspect, the present invention provides a robot inspection method for dynamic scenes. The method of the present invention is applicable to an inspection robot equipped with an autonomous navigation module (lidar and IMU), a dual-optical pan-tilt camera (a visible light camera and an infrared camera, wherein the infrared camera is used to collect infrared thermal imaging and the visible light camera is used to collect visible light images) and an edge computing unit. Specifically, Figure 1 As shown, the method of the present invention includes:

[0027] S101: Collect an infrared image of a target location and a visible light image of a candidate area.

[0028] In one embodiment, before conducting an inspection, the method of the present invention further includes: rasterizing the robot's two-dimensional map according to a specified width, drawing multiple polygonal scene areas (i.e., target locations), and configuring these scene areas as no-smoking areas. In actual applications, the target locations can be drawn and selected according to actual needs.

[0029] By locking in no-smoking areas in advance, smoking detection and false alarms in non-no-smoking areas can be avoided, thereby improving the accuracy and efficiency of smoking detection.

[0030] Specifically, when the robot is inspecting according to the inspection route, if it reaches the target location, it scans the target location through an infrared camera to detect whether there is an area that meets the temperature characteristics of the cigarette fire point (that is, the temperature is greater than the temperature threshold). If so, the area that meets the requirements is used as a candidate area, and the coordinates of the candidate fire point are marked. In one embodiment, the area with a temperature greater than 150°C is used as a candidate area. In other optional embodiments, those skilled in the art can make settings according to actual needs, for example, the area with a temperature range of 150 to 900°C is used as a candidate area.

[0031] By marking the candidate area, when performing smoking behavior detection on the candidate area of ​​the infrared image, interference from other temperatures can be avoided, thereby improving the accuracy of smoking detection based on the infrared image.

[0032] Furthermore, the visible light camera's shooting angle is adjusted based on the candidate fire point's coordinates, allowing the camera to capture a clear, centered image of the target object (i.e., the object corresponding to the candidate fire point). This image is then captured to obtain a visible light image of the candidate area. By capturing only the visible light image of the candidate area, interference from features outside the candidate area with smoke detection is avoided, thereby improving smoke detection accuracy. By adjusting the visible light camera's shooting angle based on the candidate fire point's coordinates, the candidate fire point can be centered, thereby improving the accuracy of subsequent smoke detection.

[0033] S102: Determine a first confidence level and a second confidence level.

[0034] Specifically, the infrared image captured by the infrared camera is input into a preset first classifier to obtain a first confidence level, wherein the first confidence level represents the probability of the infrared camera determining that smoking behavior exists in the candidate area.

[0035] At the same time, the visible light image captured by the visible light camera is input into a preset second classifier or neural network model to obtain a second confidence level, wherein the second confidence level represents the probability of smoking activity being detected in the candidate area based on the visible light camera.

[0036] The first classifier and the second classifier may be SMV (Support Vector Machine) classifiers.

[0037] In one embodiment, the visible light image can be input into a preset YOLOv7 model to recognize the cigarette shape, smoke texture, and hand-holding action, and output a second confidence level. In other optional embodiments, those skilled in the art can select an appropriate neural network model according to actual needs.

[0038] By identifying based on cigarette shape, smoke texture, and hand-holding action, misjudgment due to a single feature (only cigarette shape, no smoke or hand-holding action) can be avoided, thereby improving the reliability of obtaining the second confidence level.

[0039] S103 : Based on the DS evidence theory, the first confidence level and the second confidence level are integrated to obtain a smoking behavior confidence level.

[0040] DS Evidence Theory, also known as trust function theory, can effectively handle uncertain information.

[0041] In one embodiment, the conflict coefficient is determined based on the first confidence level and the second confidence level. Specifically, the calculation expression of the conflict coefficient is:

[0042] K = C1(1-C2)+C2(1-C1);

[0043] Where K is the conflict coefficient, C1 is the first confidence level, and C2 is the second confidence level.

[0044] The larger the conflict coefficient, the greater the difference between the results of smoking detection based on infrared images and the results of smoking detection based on visible light images. When the conflict coefficient is 1, the first confidence level and the second confidence level cannot be fused (in practical applications, this can be avoided through preprocessing).

[0045] Furthermore, the smoking behavior confidence level is determined based on the conflict coefficient, the first confidence level, and the second confidence level. Specifically, the smoking behavior confidence level is calculated as follows:

[0046] C=1-K·C1·C2;

[0047] In the formula, C is the smoking behavior confidence, C1 is the first confidence, C2 is the second confidence, and K is the conflict coefficient.

[0048] For example, assuming that the smoking confidence level (i.e., the first confidence level) based on the infrared image output is 0.8, the non-smoking confidence level is 0.2; the smoking confidence level (i.e., the second confidence level) based on the visible light image output is 0.7, the non-smoking confidence level is 0.3, the conflict coefficient is 0.8×0.3+0.7×0.2=0.38, and the smoking behavior confidence level is 1-0.38×0.8×0.7=0.7872.

[0049] In other optional embodiments, the first confidence level and the second confidence level may be weighted and summed to obtain the smoking behavior confidence level.

[0050] S104: Determine whether there is smoking behavior based on the smoking behavior confidence level, and if so, output a warning.

[0051] Specifically, in response to the smoking behavior confidence being greater than a first threshold (set to 0.8 in this embodiment), it is determined that smoking behavior exists, and a warning is output and the smoking subject is recorded.

[0052] By fusing the detection results of visible light images and infrared images, it is possible to avoid misjudgments and missed judgments that are prone to occur with a single sensor, thereby improving the reliability of smoking detection and thus improving the reliability of robot inspections.

[0053] Furthermore, in response to the smoking behavior confidence being less than or equal to a first threshold and greater than a second threshold (in this embodiment, the second threshold is set to 0.5), smoking behavior modeling is performed based on a Markov chain to obtain a smoking behavior state transition model, and whether smoking behavior exists is determined based on the smoking behavior state transition model.

[0054] Specifically, the smoking process is defined as a Markov chain, and the states of the smoking behavior state transition model include lighting a cigarette, holding a cigarette, exhaling smoke, and ending a cigarette, that is, S = {lighting a cigarette, holding a cigarette, exhaling smoke, ending a cigarette}; the state transition probability matrix of the smoking behavior state transition model can be obtained through historical data training.

[0055] Furthermore, the captured visible light image or video of the candidate area is input into the smoking behavior state transition model. If the smoking probability detected in three consecutive frames of visible light images is greater than a preset threshold (set to 0.6 in this embodiment), smoking behavior is determined to exist in the candidate area, and a warning is output to the smoker. The smoking probability is determined by a first probability and / or a second probability. The first probability represents the probability of transitioning from the ignition state to the holding state, and the second probability represents the probability of transitioning from the holding state to the puffing state.

[0056] Understandably, if the puffing state is detected but the holding state is not, it indicates the presence of other environmental interference such as steam (i.e., no smoking behavior in the candidate area); only a fire point (corresponding to the lighting or holding state) without smoke (corresponding to the puffing state) indicates interference such as a lighter. By dynamically modeling smoking behavior, it is possible to eliminate misjudgments caused by environmental interference, thereby improving the accuracy of smoking detection.

[0057] In one embodiment, the calculation expression of smoking probability is:

[0058] P = P1·P2;

[0059] Where P is the probability of smoking, P1 is the first probability, and P2 is the second probability.

[0060] In another embodiment, the calculation expression of the smoking probability is:

[0061] P = αP1 + βP2;

[0062] Where P is the probability of smoking, P1 is the first probability, P2 is the second probability, α is the weight of the first probability, β is the weight of the second probability, and α+β=1.

[0063] In one embodiment, α is set to 0.4 and β is set to 0.6.

[0064] By dynamically modeling smoking behavior in candidate areas where the smoking behavior confidence level is lower than 0.8, environmental interference and insufficient accuracy of static analysis can be avoided, thereby further improving the accuracy of smoking detection.

[0065] In one embodiment, before performing smoking behavior modeling, the method further includes: determining an optimal observation distance based on the smoking behavior confidence, and moving based on the optimal observation distance, wherein the optimal observation distance is negatively correlated with the smoking behavior confidence. Specifically, the calculation expression of the optimal observation distance is:

[0066] d=d min +(d max -d min )(1-C);

[0067] Where d is the optimal observation distance, d min is the preset first distance, d max The second distance is preset, d min <d max , C is the confidence level of the smoking behavior.

[0068] Among them, the first distance represents the minimum safe distance, and the second distance represents the maximum effective detection distance. In this embodiment, the first distance is set to 1 meter and the second detection distance is set to 5 meters. In other optional embodiments, those skilled in the art can set it according to actual needs.

[0069] Furthermore, when moving based on the optimal observation distance, the method includes planning an obstacle avoidance path based on the smoking behavior confidence and the environment map (generated by SLAM), wherein the cost function of the path planning is:

[0070] Cost=W1·C+W2·Obstacle+W3·Δθ;

[0071] Where Cost is the cost of path planning, Obstacle is the distance between the robot and the obstacle, and Δθ is the angle the robot needs to rotate. W1, W2, and W3 are weight coefficients, where W1 + W2 + W3 = 1.

[0072] In one embodiment, W1 is set to 0.5, W2 is set to 0.3, and W3 is set to 0.2.

[0073] In an optional embodiment, the movement may be performed according to the optimal observation distance after the smoking behavior modeling is performed.

[0074] By minimizing the cost function of this path planning, the robot's rotation angle can be dynamically adjusted to improve its dynamic obstacle avoidance capabilities. By dynamically adjusting the optimal observation distance before dynamic modeling, the system automatically expands the detection range and verifies suspicious targets from multiple angles, improving its adaptability to dynamic environments and the accuracy of smoke detection.

[0075] After smoking behavior is detected, the alarm information is transmitted back via 5G, and the timestamp, location and image evidence are stored. The alarm method can be a voice prompt.

[0076] In one embodiment, the method of the present invention further includes: determining the frequency of inspections of each target location based on the frequency of smoking detected in each target location. Specifically, the number of times smoking is detected in each target location is counted over a certain period of time (e.g., one week, one month), and the inspection frequency of the target location is determined based on the range of the number of times (e.g., once every 12 hours, once a day, once every three days, once a week, etc.). It is understood that the higher the number of smoking activities detected in a location, the higher the frequency of inspections of that location.

[0077] The following takes a campus scenario as an example to illustrate the specific application of the present invention in detail.

[0078] The inspection robot begins inspections on campus according to the set inspection route. During the driving process, its image acquisition module continuously captures images of the surrounding environment.

[0079] When the robot reaches a no-smoking area on campus (i.e., the target location), it first detects the presence of a candidate area using an infrared camera. If so, it obtains a first confidence score for the infrared image and a second confidence score for the visible light image of the candidate area. The first and second confidence scores are then combined to obtain a smoking behavior confidence score. If the smoking behavior confidence score is greater than 0.8, the robot first issues a voice warning, informing the smoker, "This is a no-smoking area. Please extinguish your cigarette immediately." If the smoker ignores the robot's warning, the robot transmits the real-time image to the backend management module, where the backend staff communicates and warns the smoker via remote communication equipment, while also recording relevant information about the smoking behavior (location, time, image, etc.).

[0080] If the confidence level of the smoking behavior is greater than 0.5 and less than or equal to 0.8, dynamic behavior modeling of the smoking behavior is performed based on the Markov chain to determine whether there are three consecutive frames of visible light images with a smoking probability greater than 0.6 in the visible light image / video input to the model. If so, it is determined that smoking behavior exists, and the smoker is warned and recorded; if not, smoking detection is carried out at the next location according to the inspection route.

[0081] If the confidence level of the smoking behavior is less than or equal to 0.5, it indicates that there is no smoking behavior in the area, and smoking detection is conducted on the next location according to the inspection route.

[0082] Throughout the inspection process, the system dynamically adjusts inspection tasks based on the characteristics of different campus areas and historical inspection data. For example, if smoking violations are common in a lawn area near a teaching building, the inspection frequency in that area will be automatically increased. When passing through that area, the robot will focus on smoking to promptly detect and address any issues, thereby improving the overall environmental quality and management efficiency of the campus.

[0083] Figure 2Schematically illustrates a structural block diagram of a robot inspection system for a dynamic scenario according to this embodiment.

[0084] In a second aspect, the present invention also provides a robot inspection system for dynamic scenes. Figure 2 As shown, the robot inspection system includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a robot inspection method for dynamic scenes is implemented.

[0085] The robot inspection system also includes other components well known to those skilled in the art, such as a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

[0086] In the present invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, the computer-readable storage medium can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store the required information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible or connectable to a device. Any application or module described in the present invention can be implemented using computer-readable / executable instructions that can be stored or otherwise retained by such a computer-readable medium.

[0087] In this specification, "multiple" means at least two, such as two, three, or more, unless otherwise specifically defined. Furthermore, the steps of the above method are divided for clarity of description only. During implementation, they can be combined into a single step, or some steps can be split into multiple steps, as long as they share the same logical relationship.

[0088] While several embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, variations, and alternatives will occur to those skilled in the art without departing from the concept and spirit of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A robot inspection method for dynamic scenes, characterized in that: include: Collecting an infrared image of the target location, and selecting areas in the infrared image where the temperature is greater than a temperature threshold as candidate areas; Acquiring a visible light image of the candidate area; Inputting the infrared image into a preset first classifier to obtain a first confidence level; Inputting the visible light image into a preset second classifier or neural network model to obtain a second confidence level; The first confidence level and the second confidence level are integrated based on the DS evidence theory to obtain the smoking behavior confidence level; In response to the smoking behavior confidence being greater than a first threshold, determining that smoking behavior exists and outputting a warning; In response to the smoking behavior confidence being less than or equal to a first threshold and greater than a second threshold, smoking behavior modeling is performed based on a Markov chain to further determine whether smoking behavior exists.

2. The robot inspection method for dynamic scenes according to claim 1 is characterized in that: The first confidence level and the second confidence level are integrated based on the DS evidence theory, including: determining a conflict coefficient between the first confidence level and the second confidence level; The smoking behavior confidence level is determined based on the conflict coefficient, the first confidence level, and the second confidence level. The smoking behavior confidence level is negatively correlated with the conflict coefficient, the first confidence level, and the second confidence level, respectively.

3. The robot inspection method for dynamic scenes according to claim 2, characterized in that: The calculation expression of the smoking behavior confidence is: C=1-K·C1·C2; In the formula, C is the smoking behavior confidence, C1 is the first confidence, C2 is the second confidence, and K is the conflict coefficient.

4. The robot inspection method for dynamic scenes according to claim 1, characterized in that: Smoking behavior modeling based on Markov chains includes: defining smoking states including lighting, holding, exhaling, and ending; obtaining a state transition probability matrix based on historical data training; Multiple frames of continuous visible light images are acquired, and if the smoking probability of the multiple frames of visible light images is greater than a preset threshold, it is determined that smoking behavior has occurred; the smoking probability is determined by a first probability and / or a second probability, the first probability representing the probability of transitioning from the ignition state to the cigarette holding state, and the second probability representing the probability of transitioning from the cigarette holding state to the puffing state.

5. The robot inspection method for dynamic scenes according to claim 4 is characterized in that: The method for determining the smoking probability includes: multiplying the first probability by the second probability to obtain the smoking probability.

6. The robot inspection method for dynamic scenes according to claim 1, characterized in that: Before performing smoking behavior modeling based on the Markov chain, the method further includes: determining an optimal observation distance according to the smoking behavior confidence, and moving based on the optimal observation distance, wherein the optimal observation distance is negatively correlated with the smoking behavior confidence.

7. The robot inspection method for dynamic scenes according to claim 6, characterized in that: The calculation expression of the optimal observation distance is: d=d min +(d max -d min )(1-C); Where d is the optimal observation distance, d min is the preset first distance, d max is the preset second distance, d min <d max , C is the confidence level of the smoking behavior.

8. The robot inspection method for dynamic scenes according to claim 6, characterized in that: Moving based on the optimal observation distance further includes: generating an obstacle avoidance path according to minimizing a cost function; the cost function is obtained by weighted summation of the smoking behavior confidence, the distance to the obstacle, and the angle required to rotate; Movement is performed based on the obstacle avoidance path.

9. The robot inspection method for dynamic scenes according to claim 1, characterized in that: Also includes: The frequency of inspections at each target location is determined based on the frequency with which smoking behavior is detected at each target location.

10. A robot inspection system for dynamic scenes, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a robot inspection method for dynamic scenes according to any one of claims 1 to 9 is implemented.

Images