Method and apparatus for detecting behavior of robot

Abstract

The application discloses a kind of robot behavior detection method and device. Among them, the method comprises: obtaining environmental parameter data and image data;The effective environmental data of the same scale is obtained by processing environmental parameter data, while the first image data and the second image data are cropped to obtain the third image data and the fourth image data related to the target robot;The comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data and the fourth image data is determined by a feature fusion model, wherein the convolutional neural network model of the feature fusion model is used to process the third image data and the fourth image data, and the full connection layer of the feature fusion model is used to process the effective environmental data;The behavior data of the target robot is obtained according to the comprehensive feature data.The application solves the technical problem that a single information source is difficult to accurately determine the behavior of the robot in a complex environment in the related art, and misjudgment and omission are prone to occur.

Description

Technical Field

[0001] This invention relates to the field of inspection robot behavior detection, and more specifically, to a robot behavior detection method and apparatus. Background Technology

[0002] With the development of industrial automation and intelligence, inspection robots are being used more and more widely in various scenarios. Traditional methods for detecting the behavior of inspection robots often rely solely on information from the sensors carried by the robot itself, which limits the accuracy and comprehensiveness of the detection. At the same time, in complex environments, a single information source is insufficient to accurately determine the robot's behavior, easily leading to misjudgments and missed detections.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a method and apparatus for detecting robot behavior, which at least solves the technical problem in related technologies that in complex environments, a single information source is difficult to accurately determine the behavior of a robot, and misjudgments and omissions are prone to occur.

[0005] According to one aspect of the present invention, a method for detecting robot behavior is provided, comprising: acquiring environmental parameter data and image data, wherein the environmental parameter data is data collected by an information sensing component, the image data includes first image data and second image data, the first image data is data collected by a first image acquisition component, the second image data is data collected by a second image acquisition component, the information sensing component and the first image acquisition component are mounted on a target robot, the second image acquisition component is disposed in the area where the target robot is located, and the target robot is a robot for which behavior detection is required; normalizing the environmental parameter data to obtain effective environmental data of the same scale, and simultaneously cropping the first image data and the second image data to obtain third image data and fourth image data related to the target robot; determining comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data through a feature fusion model, wherein the convolutional neural network model of the feature fusion model is used to process the third image data and the fourth image data, the fully connected layer of the feature fusion model is used to process the effective environmental data, and the feature fusion model is a pre-trained model; and obtaining behavior data of the target robot based on the comprehensive feature data.

[0006] Optionally, acquiring environmental parameter data and image data includes: triggering the information sensing component to collect the environmental parameter data around the target robot while the target robot moves along the inspection route, and simultaneously triggering the first image acquisition component to collect the first image data around the target robot and the second image acquisition component to collect the second image data of the area where the target robot is located.

[0007] Optionally, normalizing the environmental parameter data to obtain valid environmental data of the same scale includes: performing anomaly detection on the environmental parameter data to obtain abnormal data in the environmental parameter data; processing the abnormal data to obtain processed environmental parameter data; and normalizing the processed environmental parameter data to obtain valid environmental data of the same scale.

[0008] Optionally, the first image data and the second image data are cropped to obtain the third image data and the fourth image data related to the target robot, including: cropping the background area in the first image data according to the first viewing angle range of the target robot and monitoring requirement information to obtain the third image data related to the target robot; and cropping the non-key monitoring area in the second image data that is not related to the target robot according to the monitoring requirement information and the second viewing angle range of the second image acquisition component to obtain the fourth image data related to the target robot.

[0009] Optionally, before acquiring environmental parameter data and image data, the robot's behavior detection method further includes: calibrating the information sensing component using historical temperature data and historical humidity data, wherein the historical temperature data and historical humidity data are data collected by the information sensing component within a historical time period; and calibrating the first image acquisition component according to an intrinsic parameter matrix and an extrinsic parameter matrix, wherein the intrinsic parameter matrix and the extrinsic parameter matrix are calculated based on sample images from different angles acquired by the first image acquisition component.

[0010] Optionally, before acquiring environmental parameter data and image data, the robot's behavior detection method further includes: initializing the target robot's operating system and data acquisition program to set the data acquisition frequency and data storage path of the information sensing component and the first image acquisition component; initializing the second image acquisition component to set the image acquisition parameters and image acquisition range of the second image acquisition component, wherein the image acquisition range is associated with the map corresponding to the monitoring area of ​​the second image acquisition component.

[0011] Optionally, determining the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data through a feature fusion model includes: in the feature fusion model, using the convolutional neural network model to extract features from the third image data and the fourth image data to obtain the first image features and the second image features of the target robot, and using the fully connected layer to extract features from the effective environmental data to obtain environmental parameter features; and using the fully connected layer to concatenate the first image features, the second image features, and the environmental parameter features to obtain the comprehensive feature data.

[0012] Optionally, when the inspection area of ​​the target robot includes surrounding robots, the surrounding robots are robots in the same inspection area as the target robot. The behavior detection method for these robots further includes: acquiring surrounding environmental parameter data and surrounding image data of the surrounding robots; integrating the environmental parameter data, the image data, the surrounding environmental parameter data, and the surrounding image data to obtain integrated information; processing the integrated information using a distributed prediction model to obtain a behavior collaborative determination result for the target robot, wherein the distributed prediction model is pre-trained based on historical data, and the historical data includes the following information about the target robot: movement direction, speed change, and surrounding obstacle status; and obtaining the behavior data of the target robot based on the collaborative determination result.

[0013] Optionally, after obtaining the behavior data of the target robot based on the collaborative determination result, the robot behavior detection method further includes: updating the behavior data using the collaborative determination result.

[0014] Optionally, the robot behavior detection method further includes: acquiring environmental change information of the environment in which the target robot is located; updating the parameters of the image processing algorithm according to the environmental change information, wherein the image processing algorithm is used to process the first image data and the second image data.

[0015] Optionally, the robot's behavior detection method further includes: determining the behavior pattern corresponding to the behavior data through a behavior warning model, wherein the behavior warning model is obtained by machine learning training using multiple sets of training data, historical behavior data is the input of the behavior warning model, and the historical behavior pattern corresponding to the historical behavior data is the output of the behavior warning model.

[0016] Optionally, the robot behavior detection method further includes: when the current temperature of a predetermined component of the target robot is detected to be higher than a predetermined temperature, determining the temperature information of the predetermined component through a knowledge graph, wherein the temperature information includes the current temperature and the ambient temperature of the predetermined component, and the knowledge graph is a graph pre-constructed based on the various components of the target robot and the environment in which the target robot is located; and determining the fault factors and fault handling methods of the predetermined component based on the current temperature and the ambient temperature.

[0017] According to another aspect of the present invention, a robot behavior detection device is also provided, comprising: a first acquisition unit, configured to acquire environmental parameter data and image data, wherein the environmental parameter data is data collected by an information sensing component, and the image data includes first image data and second image data, wherein the first image data is data collected by a first image acquisition component, and the second image data is data collected by a second image acquisition component, the information sensing component and the first image acquisition component are mounted on a target robot, and the second image acquisition component is disposed in the area where the target robot is located, and the target robot is a robot for which behavior detection is required; and a processing unit, configured to perform normalization processing on the environmental parameter data to obtain... The first image data and the second image data are cropped simultaneously to obtain third image data and fourth image data related to the target robot; a first determining unit is used to determine the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data and the fourth image data through a feature fusion model, wherein the convolutional neural network model of the feature fusion model is used to process the third image data and the fourth image data, the fully connected layer of the feature fusion model is used to process the effective environmental data, and the feature fusion model is a pre-trained model; a second acquiring unit is used to obtain the behavior data of the target robot based on the comprehensive feature data.

[0018] Optionally, the first acquisition unit includes: a triggering module, configured to trigger the information sensing component to collect the environmental parameter data around the target robot during the process of the target robot moving along the inspection route, and simultaneously trigger the first image acquisition component to collect the first image data around the target robot and the second image acquisition component to collect the second image data of the area where the target robot is located.

[0019] Optionally, the processing unit includes: a detection module for detecting anomalies in the environmental parameter data to obtain abnormal data in the environmental parameter data; a first processing module for processing the abnormal data to obtain processed environmental parameter data; and a second processing module for normalizing the processed environmental parameter data to obtain valid environmental data of the same scale.

[0020] Optionally, the processing unit includes: a first cropping module, configured to crop the background area in the first image data according to the first viewing angle range of the target robot and monitoring requirement information to obtain the third image data related to the target robot; and a second cropping module, configured to crop the non-key monitoring area in the second image data that is not related to the target robot according to the monitoring requirement information and the second viewing angle range of the second image acquisition component to obtain the fourth image data related to the target robot.

[0021] Optionally, the robot's behavior detection device further includes: a first calibration unit, used to calibrate the information sensing component using historical temperature data and historical humidity data before acquiring environmental parameter data and image data, wherein the historical temperature data and historical humidity data are data collected by the information sensing component within a historical time period; and a second calibration unit, used to calibrate the first image acquisition component according to an intrinsic parameter matrix and an extrinsic parameter matrix, wherein the intrinsic parameter matrix and the extrinsic parameter matrix are calculated based on sample images acquired by the first image acquisition component from different angles.

[0022] Optionally, the robot's behavior detection device further includes: a first initialization unit, used to initialize the target robot's operating system and data acquisition program before acquiring environmental parameter data and image data, to set the data acquisition frequency and data storage path of the information sensing component and the first image acquisition component; and a second initialization unit, used to initialize the second image acquisition component, to set the image acquisition parameters and image acquisition range of the second image acquisition component, wherein the image acquisition range is associated with a map corresponding to the monitoring area of ​​the second image acquisition component.

[0023] Optionally, the first determining unit includes: an extraction module, configured to use the convolutional neural network model to extract features from the third image data and the fourth image data to obtain first image features and second image features of the target robot, and use the fully connected layer to extract features from the effective environmental data to obtain environmental parameter features; and a stitching module, configured to use the fully connected layer to stitch together the first image features, the second image features, and the environmental parameter features to obtain the comprehensive feature data.

[0024] Optionally, the robot's behavior detection device further includes: a third acquisition unit, configured to acquire surrounding environmental parameter data and surrounding image data of the surrounding robots when the inspection area of ​​the target robot includes surrounding robots, wherein the surrounding robots are robots in the same inspection area as the target robot; an integration unit, configured to integrate the environmental parameter data, the image data, the surrounding environmental parameter data, and the surrounding image data to obtain integrated information; a fourth acquisition unit, configured to process the integrated information through a distributed prediction model to obtain the behavior collaborative judgment result of the target robot, wherein the distributed prediction model is pre-trained based on historical data, and the historical data includes the following information of the target robot: movement direction, speed change, and surrounding obstacle status; and a fifth acquisition unit, configured to obtain the behavior data of the target robot based on the collaborative judgment result.

[0025] Optionally, the robot's behavior detection device further includes: a first update unit, used to update the behavior data using the collaborative determination result after obtaining the behavior data of the target robot based on the collaborative determination result.

[0026] Optionally, the robot's behavior detection device further includes: a sixth acquisition unit, used to acquire environmental change information of the environment in which the target robot is located; and a second update unit, used to update the parameters of the image processing algorithm according to the environmental change information, wherein the image processing algorithm is used to process the first image data and the second image data.

[0027] Optionally, the robot's behavior detection device further includes: a second determining unit, used to determine the behavior pattern corresponding to the behavior data through a behavior warning model, wherein the behavior warning model is obtained by machine learning training using multiple sets of training data, historical behavior data is the input of the behavior warning model, and the historical behavior pattern corresponding to the historical behavior data is the output of the behavior warning model.

[0028] Optionally, the robot's behavior detection device further includes: a third determining unit, configured to determine temperature information of the predetermined component through a knowledge graph when the current temperature of a predetermined component of the target robot is detected to be higher than a predetermined temperature, wherein the temperature information includes the current temperature and the ambient temperature of the predetermined component, and the knowledge graph is a graph pre-constructed based on the various components of the target robot and the environment in which the target robot is located; and a fourth determining unit, configured to determine the fault factors and fault handling methods of the predetermined component based on the current temperature and the ambient temperature.

[0029] According to another aspect of the present invention, a robot behavior detection system is also provided, which uses the robot behavior detection method described in any one of the above embodiments.

[0030] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes the robot behavior detection method described in any one of the foregoing embodiments.

[0031] According to another aspect of the present invention, a processor is also provided, the processor being configured to run a program, wherein the program, when running, executes the robot behavior detection method described in any of the foregoing embodiments.

[0032] According to another aspect of the present invention, a computer program product is also provided, including computer instructions that, when executed by a processor, perform the robot behavior detection method described in any one of the above embodiments.

[0033] In this embodiment of the invention, environmental parameter data and image data are acquired. The environmental parameter data is data collected by an information sensing component, and the image data includes first image data and second image data. The first image data is data collected by a first image acquisition component, and the second image data is data collected by a second image acquisition component. The information sensing component and the first image acquisition component are mounted on a target robot, and the second image acquisition component is located in the area where the target robot is located. The target robot is a robot for which behavior detection is required. The environmental parameter data is normalized to obtain effective environmental data of the same scale. At the same time, the first and second image data are cropped to obtain third and fourth image data related to the target robot. The comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data are determined by a feature fusion model. The convolutional neural network model of the feature fusion model is used to process the third and fourth image data, and the fully connected layer of the feature fusion model is used to process the effective environmental data. The feature fusion model is a pre-trained model. The behavior data of the target robot is obtained based on the comprehensive feature data. The above technical solution achieves the goal of integrating the robot's own sensor data, its own image data, and the image data from external monitors. By obtaining more comprehensive information from multiple data sources, the robot's behavior detection is improved, which enhances the accuracy, robustness, and robustness of the robot's behavior detection. It also reduces the risk of misjudgment caused by a single data source. This solves the technical problem in related technologies where a single information source is difficult to accurately determine the robot's behavior in complex environments, and is prone to misjudgment and missed judgment. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0035] Figure 1 This is a hardware structure block diagram of a mobile terminal for a robot behavior detection method according to an embodiment of the present invention.

[0036] Figure 2 This is a flowchart of a robot behavior detection method according to an embodiment of the present invention;

[0037] Figure 3 This is a flowchart of data collection and processing according to an embodiment of the present invention;

[0038] Figure 4 This is a flowchart of the construction and training of a dual-channel fusion training model according to an embodiment of the present invention;

[0039] Figure 5 This is a framework diagram of the integrated adaptation technology for the robot behavior detection method according to the present invention;

[0040] Figure 6 This is a schematic diagram of a robot behavior detection device according to an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] As described in the background section, in complex environments, a single information source is insufficient to accurately determine robot behavior, leading to frequent misjudgments and missed detections. For example, in the automotive manufacturing industry, painting robots are used for paint application to improve production efficiency. However, abnormalities during operation can affect the painting effect and vehicle body quality. Furthermore, existing technologies rely on manual inspection for anomaly detection, which is difficult, time-consuming, and prone to errors. To address this deficiency, existing technologies collect vehicle body information (including VIN, vehicle number, etc.) and painting robot operating data (including actual and set values ​​of operating parameters) in real time for each production line. The operating data is used to determine whether the painting robot is working properly. For each operating parameter of each painting robot, if the actual value deviates significantly from the set value in a preset number of data collection cycles (e.g., 20 times), the painting robot is considered not working properly. However, this approach has the following drawbacks: 1) Static threshold dependence: It relies entirely on preset fixed thresholds (set values) for anomaly detection, failing to address dynamic changes in the production environment. For example, changes in operating conditions such as coating viscosity and temperature may cause parameters to drift within a reasonable range, but the system cannot automatically adjust the threshold, resulting in missed detections or misjudgments; 2) Isolated single-parameter analysis: Only a single operating parameter (such as high pressure, speed) is judged in isolation, ignoring the coupling relationship between parameters. For example, high pressure anomalies may occur together with flow parameter anomalies, but this method cannot discover composite fault modes through multi-dimensional data analysis; 3) Insufficient capture of temporal features: Using discrete sampling with a fixed window (such as 20 consecutive samples) makes it difficult to identify periodic fluctuations or gradual anomalies. For example, the spraying pressure may slowly decrease over time, which may have a cumulative impact on coating quality before reaching the alarm threshold; 4) Lack of fault tracing information: Although abnormal parameters can be located, there is a lack of correlation analysis with the physical state of the equipment. For example, only "high pressure anomaly" is indicated, but it is impossible to distinguish whether it is a sensor failure, pipeline blockage, or motor performance degradation, increasing the difficulty of maintenance; 5) Weak data-driven capability: It relies entirely on a preset rule base and does not introduce machine learning algorithms. The inability to mine abnormal patterns from historical data to build a fault prediction model limits the system's upgrade from passive detection to proactive early warning; 5) Insufficient trade-off between real-time performance and accuracy: Using sliding window mean calculation (e.g., 20 consecutive samples) sacrifices real-time performance for stability, but in sudden fault scenarios (e.g., power outage), alarms may be delayed by several minutes, affecting emergency response efficiency. These defects result in insufficient robustness of the system under complex operating conditions, making it difficult to meet the high precision and strong adaptability requirements of intelligent manufacturing for anomaly detection systems. There is an urgent need to introduce advanced technologies such as dynamic threshold adjustment, multimodal perception fusion, and digital twins for optimization and upgrading.

[0044] For example, existing technologies perform anomaly detection on robots in the following ways: 1) Data acquisition and storage: The robot actively reports its own behavior and status data and uploads it to a server (such as a Kafka cluster). The server processes the data in real time and stores it in a database (such as MySQL or HDFS). 2) Anomaly detection: Non-real-time detection: Robot data is aggregated at set time intervals (such as calculating the highest CPU usage value every day), and anomalies are analyzed after the set detection period ends. Real-time detection: Based on set rules (such as the number of consecutive falls within 1 minute), the system judges whether the robot is abnormal in real time. 3) Anomaly handling: When an anomaly is detected, the system sends an alert (SMS, email) to preset contacts; the anomaly detection results are stored in the database for querying and analysis; the server can send remote debugging commands to the robot (such as terminating processes with high CPU usage). 4) Dynamic rule updates: The server can periodically or when rule changes are detected to obtain the latest anomaly detection rules to ensure the accuracy of the detection logic. However, the above technical solutions have the following drawbacks: 1) Single information source and lack of multi-source fusion: This patent relies solely on the robot's own reported data for anomaly detection, while this invention uses dual-channel training, combining data from the robot's own sensors and external monitors to provide more comprehensive detection results and reduce missed and false positives; 2) Lack of swarm intelligence and collaborative judgment: This invention designs an information sharing and collaborative judgment mechanism among robots, using Bayesian networks and distributed collaborative prediction algorithms to allow multiple robots to mutually verify anomalies, improving detection accuracy. This patent, however, only performs individual robot detection, lacking a collaborative mechanism between robots, making it difficult to adapt to false positives in complex environments; 3) Lack of dynamic environment adaptability: This invention introduces dynamic environment adaptive technology, which can automatically adjust detection parameters according to changes in environment such as light and temperature, improving robustness. The lack of similar technology here may lead to decreased accuracy under different environmental conditions; 4) Lack of abnormal behavior prediction capabilities: While it utilizes deep learning algorithms such as LSTM for behavior prediction and early warning, enabling early detection of potential faults, it relies on rule matching and primarily detects already occurring anomalies, lacking prediction and early warning functions; 5) Lack of semantic understanding and knowledge graph support: Through knowledge graphs and semantic analysis, it can understand the underlying causes of anomalies and correlate them with equipment characteristics, potential faults, and handling measures, improving intelligent decision-making capabilities. This system, however, only relies on numerical statistics for anomaly detection and cannot provide in-depth fault cause analysis and intelligent suggestions.

[0045] The present invention provides a robot behavior detection method and apparatus, a robot behavior detection system, a computer-readable storage medium, a processor, and a computer program product.

[0046] Furthermore, the technical solutions provided in this invention rely on the following technologies: 1) Robot end sensor technology, including various sensors used to acquire device status (such as vibration sensors, current sensors, etc.) and environmental parameters (such as temperature, humidity, gas concentration sensors, etc.), and cameras used to acquire image information (such as wide-angle cameras, high-definition cameras, etc.); 2) External monitoring technology, such as high-resolution, wide-angle monitoring cameras, and related parameter settings (resolution, frame rate, exposure time, etc.) and image acquisition range delineation technology; 3) Robot sensor calibration technology, such as high-precision calibration methods for temperature sensors, humidity sensors, etc., and calibration technologies such as Zhang's calibration method for cameras; 4) Image preprocessing technology, including image cropping (removing irrelevant backgrounds and retaining key areas) and normalization (linear normalization, standardization, etc.) operations; 5) Multimodal alignment technology, used to align images acquired by the robot, environmental parameter data, and image data acquired by the external monitor, so that data from different modalities can be effectively fused.

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0048] The methods and embodiments provided in this invention can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a robot behavior detection method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0049] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the robot behavior detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0050] Example 1

[0051] According to an embodiment of the present invention, a method embodiment for detecting the behavior of a robot is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0052] Figure 2 This is a flowchart of a robot behavior detection method according to an embodiment of the present invention, such as... Figure 2 As shown, the robot's behavior detection method includes the following steps:

[0053] Step S202: Obtain environmental parameter data and image data. The environmental parameter data is data collected by the information sensing component. The image data includes first image data and second image data. The first image data is data collected by the first image acquisition component, and the second image data is data collected by the second image acquisition component. The information sensing component and the first image acquisition component are mounted on the target robot, and the second image acquisition component is set in the area where the target robot is located. The target robot is a robot that needs to perform behavior detection.

[0054] Optionally, the target robot mentioned above can be an inspection robot.

[0055] Optionally, the aforementioned information sensing component can be various sensors (e.g., temperature sensor, humidity sensor, etc.); the aforementioned first image acquisition component and second image acquisition component can be cameras, where the cameras are divided into a first image acquisition component mounted on the target robot and a second image acquisition component (external monitor) set in the area where the target robot is located.

[0056] Optionally, the area where the target robot is located refers to the inspection area of ​​the target robot, or it can be an area with the target robot as the center and a predetermined length as the radius. There is no specific limitation here, and it can be any area where the robot may appear.

[0057] For example, the aforementioned external monitor could be a monitor installed within the inspection area of ​​the target robot.

[0058] Optionally, the aforementioned environmental parameter data may include, but is not limited to, temperature, humidity, and gas concentration. The aforementioned first image data and second image data may include, but are not limited to, obstacles and equipment around the robot.

[0059] Taking inspection robots as an example, the inspection robot is equipped with various types of cameras (the first image acquisition component) (such as wide-angle cameras for capturing large-scale scenes and high-definition cameras for capturing details) and various sensors (such as temperature sensors, humidity sensors, vibration sensors, etc.). These devices are used to perceive the surrounding environment in real time, covering the equipment's operating status (such as equipment temperature, vibration frequency, and other parameters) and environmental parameters (such as temperature, humidity, gas concentration, etc.).

[0060] The aforementioned external monitoring devices can be deployed reasonably within the monitoring area. These devices are characterized by high resolution and wide viewing angle, and are used to synchronously collect video image information of the corresponding area (second image acquisition component) to provide the system with environmental scene data from different perspectives.

[0061] Step S204: Normalize the environmental parameter data to obtain effective environmental data of the same scale. At the same time, crop the first image data and the second image data to obtain the third image data and the fourth image data related to the target robot.

[0062] In this embodiment, the environmental parameter data can be normalized to obtain effective environmental data of the same scale; in addition, the first image data and the second image data can be processed to obtain the part of the first image data with the background removed (i.e., the third image data) and the part of the second image data including the target robot (i.e., the fourth image data).

[0063] Step S206: Determine the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data through the feature fusion model. The convolutional neural network model of the feature fusion model is used to process the third image data and the fourth image data, and the fully connected layer of the feature fusion model is used to process the effective environmental data. The feature fusion model is a pre-trained model.

[0064] In this embodiment, image data is processed by a convolutional neural network (CNN) to extract image features; environmental parameter data is processed by a fully connected layer to extract parameter features; in the feature fusion model, an attention mechanism is used to weight different modal features to improve the expressive power of the features; image features, environmental parameter features and monitor image features are concatenated and further fused through multiple fully connected layers to form a comprehensive feature representation.

[0065] For example, outlier filtering and normalization are performed on environmental parameter data to ensure data reliability; image data is cropped and normalized to remove irrelevant background and retain key areas. These will be described in detail below and will not be repeated here.

[0066] Step S208: Obtain the target robot's behavior data based on the comprehensive feature data.

[0067] In this embodiment, the fused features are input into a pre-trained model to output behavioral data, thereby achieving accurate detection of robot behavior.

[0068] As described above, in this embodiment of the invention, environmental parameter data and image data can be acquired. The environmental parameter data is data collected by the information sensing component, and the image data includes first image data and second image data. The first image data is data collected by the first image acquisition component, and the second image data is data collected by the second image acquisition component. The information sensing component and the first image acquisition component are mounted on the target robot, and the second image acquisition component is located in the area where the target robot is located. The target robot is a robot that requires behavior detection. Next, the environmental parameter data is normalized to obtain effective environmental data of the same scale. Simultaneously, the first and second image data are cropped to obtain third and fourth image data related to the target robot. The feature fusion model determines the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data. The convolutional neural network model of the feature fusion model is used to process the third and fourth image data, and the fully connected layer of the feature fusion model is used to process the effective environmental data. The feature fusion model is a pre-trained model. The behavior data of the target robot is obtained based on the comprehensive feature data. This achieves the goal of fusing the robot's own sensor data, its own image data, and the image data of the external monitor. By obtaining more comprehensive information from multiple data sources, the robot's behavior detection is improved, which improves the accuracy, robustness, and accuracy of robot behavior detection and reduces the risk of misjudgment caused by a single data source.

[0069] Therefore, the technical solutions provided by the embodiments of the present invention solve the technical problem in the related art that in complex environments, it is difficult for a single information source to accurately determine the behavior of a robot, which easily leads to misjudgment and omission.

[0070] According to the above embodiments of the present invention, acquiring environmental parameter data and image data may include: triggering an information sensing component to collect environmental parameter data around the target robot while the target robot moves along the inspection route, and simultaneously triggering a first image acquisition component to collect first image data around the target robot and a second image acquisition component to collect second image data of the area where the target robot is located.

[0071] In this embodiment, when the robot (i.e., the target robot) initiates the inspection task, the data acquisition program begins to run. The robot moves along a preset inspection route. During the movement, the camera acquires environmental images (first image data) in real time at a set frame rate, and various sensors acquire environmental parameter data at a fixed frequency. The acquired data is then transmitted to the data storage module for temporary storage via the robot's internal bus.

[0072] In addition, at the external monitor's data acquisition end, the external monitor and the robot's data acquisition program start synchronously, continuously acquiring video images of the corresponding area. The monitor transmits the acquired video stream to the data storage server in real time via the network, and the storage server stores the video data in chronological order.

[0073] For example, when the target robot starts and moves along a preset route, the information sensing component and image acquisition component are triggered to begin working. The information sensing component collects environmental parameter data around the target robot in real time, such as temperature and humidity; simultaneously, the first and second image acquisition components collect image information of the surrounding environment and the area within which the target robot is located, respectively. Furthermore, the collected data is temporarily stored in the target robot's storage module and then transmitted to the central processing system via a wireless network for further processing. When integrating the data, it is ensured that the first and second image data are aligned on the timeline to facilitate subsequent fusion analysis.

[0074] Figure 3 This is a flowchart of data collection and processing according to an embodiment of the present invention, such as... Figure 3 As shown, this process can include data acquisition, data processing, and pre-acquisition preparation. This section primarily describes data acquisition, such as... Figure 3 As shown, data acquisition can include robot data acquisition (images captured by cameras and environmental parameters captured by sensors) and external monitor data acquisition (video image acquisition). The data acquired by the robot is temporarily stored in the robot's internal storage module, while the video images are temporarily stored on the server.

[0075] It should be noted that data storage format conversion will be performed during data storage, for example, such as... Figure 3 As shown, the environmental parameter data and image data collected by the robot are converted into a unified storage format: CSV (for storing environmental parameters) and JPEG (for storing images). For video data collected by an external monitor, it is segmented into single-frame images and converted to JPEG format for storage, for subsequent processing.

[0076] Real-time data acquisition ensures that the acquired data is closely related to the robot's current environment, enhancing the timeliness and accuracy of behavior detection. Temporary data storage and integration facilitate subsequent model training and behavior analysis, guaranteeing the smooth operation of the entire system.

[0077] According to the above embodiments of the present invention, normalizing environmental parameter data to obtain effective environmental data of the same scale may include: performing anomaly detection on the environmental parameter data to obtain abnormal data in the environmental parameter data; processing the abnormal data to obtain processed environmental parameter data; and normalizing the processed environmental parameter data to obtain effective environmental data of the same scale.

[0078] In this embodiment, such as Figure 3 As shown, outlier detection and processing are performed on the collected environmental parameter data. Statistical methods are used to detect outliers in the data, and corrections or removals are made based on the actual situation. Simultaneously, environmental parameters are normalized to ensure they have the same scale range as other data features.

[0079] For example, anomaly detection is performed on the collected environmental parameter data. Temperature data below 0℃ or above 50℃, and humidity data below 30% or above 80% are considered abnormal and need to be removed or corrected. For detected outliers, interpolation is used to correct them and restore them to the normal range.

[0080] In addition, the processed environmental parameter data is normalized to a scale range of [-1,1], which facilitates the training and analysis of subsequent deep learning models.

[0081] Through the aforementioned anomaly detection and data correction, the quality of environmental parameter data can be ensured, avoiding training bias in the behavior detection model caused by outliers. Furthermore, data normalization further improves the efficiency of model training and the accuracy of detection results, enhancing the system's robustness.

[0082] According to the above embodiments of the present invention, cropping processing is performed on the first image data and the second image data to obtain third image data and fourth image data related to the target robot, including: cropping the background area in the first image data according to the first viewing angle range of the target robot and the monitoring requirement information to obtain the third image data related to the target robot; and cropping the non-key monitoring area in the second image data that is not related to the target robot according to the monitoring requirement information and the second viewing angle range of the second image acquisition component to obtain the fourth image data related to the target robot.

[0083] In this embodiment, the first image data and the second image data are cropped. For example, irrelevant backgrounds in the first image data are cropped out based on the first viewpoint range of the target robot, while retaining key areas; non-key monitoring areas in the second image data are cropped out based on monitoring requirements and the second viewpoint range.

[0084] That is, such as Figure 3 As shown, the acquired images are cropped according to the field of view of the robot and the monitor, as well as the actual monitoring needs. For images captured by the robot's camera, irrelevant background areas are cropped out, retaining the parts related to the equipment and environment; for images captured by the monitor, images of key monitoring areas are cropped out.

[0085] In addition, image normalization can be performed, that is, the cropped image is normalized (linear or standardized normalization) to ensure the scale consistency of image features, which facilitates subsequent feature extraction and fusion (comprehensive processing), and can improve the training efficiency and accuracy of deep learning models.

[0086] Image cropping and normalization can significantly reduce interference from invalid information, improve the efficiency and accuracy of image feature extraction, and thus enhance the robot's behavior detection capabilities. Simultaneously, normalization also benefits model training, avoiding training instability caused by differences in image scale.

[0087] Based on the above embodiments of the present invention, before acquiring environmental parameter data and image data, the robot behavior detection method may further include: calibrating the information sensing component using historical temperature data and historical humidity data, wherein the historical temperature data and historical humidity data are data collected by the information sensing component within a historical time period; calibrating the first image acquisition component according to the intrinsic parameter matrix and the extrinsic parameter matrix, wherein the intrinsic parameter matrix and the extrinsic parameter matrix are calculated based on sample images from different angles acquired by the first image acquisition component.

[0088] In this embodiment, information sensing components can be calibrated. For example, for temperature and humidity sensors, high-precision calibration equipment is used, and multiple measurements are performed under different standard temperature and humidity conditions according to the sensor's calibration specifications. The deviation between the measured values ​​and the standard values ​​is recorded, and the sensor output is calibrated using software algorithms to ensure the accuracy and reliability of the collected environmental parameters. Additionally, the robot's camera can be calibrated. Specifically, Zhang's calibration method is used to calibrate the camera mounted on the robot. By capturing checkerboard images from different angles, the intrinsic parameter matrix (including focal length, principal point coordinates, etc.) and extrinsic parameter matrix (including rotation and translation vectors) of the camera are calculated to correct camera distortion and improve the accuracy of image acquisition.

[0089] like Figure 3 As shown, pre-data acquisition work may include robot equipment calibration and initialization, and external monitor settings. Specifically, the information sensing component calibration involves using high-precision calibration equipment to perform multiple measurements on the information sensing component under standard conditions of 20℃±3℃ and 50%±5% humidity. The deviation between the measured values ​​and the standard values ​​is recorded, and calibration adjustments are made using software algorithms. The image acquisition component calibration employs Zhang's calibration method, using chessboard images taken from different angles to calculate the camera's intrinsic parameter matrix (including focal length, principal point coordinates, etc.) and extrinsic parameter matrix (including rotation and translation vectors) to correct camera distortion and improve image acquisition accuracy.

[0090] According to the above embodiments of the present invention, before acquiring environmental parameter data and image data, the robot behavior detection method may further include: initializing the operating system and data acquisition program of the target robot to set the data acquisition frequency and data storage path of the information sensing component and the first image acquisition component; initializing the second image acquisition component to set the image acquisition parameters and image acquisition range of the second image acquisition component, wherein the image acquisition range is associated with the map corresponding to the monitoring area of ​​the second image acquisition component.

[0091] like Figure 3 As shown, the initialization of the robot device can be achieved by initializing the robot's operating system and data acquisition program, setting parameters such as the data acquisition frequency and storage path, to ensure that the system can operate stably and accurately record the acquired data.

[0092] External monitors can be initialized in the following ways: (1) Parameter settings: According to the characteristics and needs of the monitoring area, set the monitor's resolution (e.g., 1080P, 4K, etc.), frame rate (e.g., 25fps, 30fps, etc.), exposure time, white balance, etc., to obtain high-quality video images; (2) Image acquisition range delineation: Use geographic information system (GIS) or on-site measurement to accurately delineate the image acquisition range of each monitor and associate it with the map of the monitoring area for subsequent data fusion and analysis.

[0093] For example, the operating system and data acquisition program are initialized: the data acquisition frequency is set to 10 times per second, and the storage path is the internal hard drive, ensuring stable system operation and accurate recording of acquired data. The second image acquisition component is initialized: the monitor's resolution (e.g., 1080P), frame rate (e.g., 30fps), exposure time (automatically adjusted), and white balance (automatically adjusted) are set, and the image acquisition range is precisely delineated using a Geographic Information System (GIS) or on-site measurement to ensure its matching with the monitoring area map.

[0094] Equipment calibration and initialization ensure the accuracy and consistency of data acquisition, reducing behavior detection errors caused by equipment malfunctions. Simultaneously, initialization settings lay a solid foundation for subsequent data processing and analysis, ensuring the smooth operation of the entire system.

[0095] According to the above embodiments of the present invention, determining the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data through a feature fusion model may include: in the feature fusion model, using a convolutional neural network model to extract features from the third image data and the fourth image data to obtain the first image features and the second image features of the target robot, and using a fully connected layer to extract features from the effective environmental data to obtain environmental parameter features; and using a fully connected layer to concatenate the first image features, the second image features, and the environmental parameter features to obtain the comprehensive feature data.

[0096] In this embodiment, multimodal alignment technology can be used to extract features from images acquired by the robot, environmental parameter data, and images acquired by an external monitor, respectively. For image data, a convolutional neural network (CNN) is used. For environmental parameter data, a fully connected layer is used to map it into feature vectors. An attention mechanism is used to weight the data features of different modalities, highlighting important feature information and improving the expressive power of the features. The extracted feature vectors from different modalities are then fused through fully connected layers. Specifically, robot image features, environmental parameter features, and monitor image features are concatenated, and then fused through multiple fully connected layers to obtain a comprehensive feature representation. Training is then performed. Figure 4 This is a flowchart of the construction and training of a dual-channel fusion training model according to an embodiment of the present invention, such as... Figure 4 As shown, this process can include data input, feature extraction, and feature fusion. Data input can include images captured by the robot, environmental parameter data captured by the robot, and images captured by an external monitor. The feature extraction stage can include: a CNN extracting robot image features and external monitor image features, and a fully connected layer extracting robot environmental parameter data; and outputting robot feature vectors, robot environmental parameter feature vectors, and external monitor image feature vectors respectively. In the feature fusion stage, the above feature vectors are concatenated and input into multiple fully connected layers for feature fusion, finally outputting a comprehensive feature representation (i.e., comprehensive feature data).

[0097] In this embodiment of the invention, multimodal alignment technology is used in model training to ensure the spatiotemporal consistency of data from different modalities. Historical data is used for supervised training to optimize model parameters, enabling the model to effectively analyze multi-source information.

[0098] By constructing a feature fusion model, image information and environmental information are effectively combined, improving the accuracy and robustness of behavior detection. Simultaneously, the application of deep learning technology enables the automatic learning and extraction of more complex features, enhancing the model's generalization ability.

[0099] The dual-channel training mode here involves two aspects. On one hand, the inspection robot acquires information about its surrounding environment, including equipment status and environmental parameters, through its onboard cameras, sensors, and other equipment. On the other hand, external monitors set up within the monitoring area simultaneously collect video images of the corresponding areas.

[0100] Furthermore, deep learning algorithms are used to fuse and train a model that integrates information collected by the robot itself and that collected by the monitor. Multimodal alignment technology is employed to extract features from images, videos, and statistical data from both information sources. These extracted features are then fused to train a model capable of comprehensively analyzing multi-source information. In this way, the robot can not only understand its environment from its own perspective but also leverage the global perspective information from the external monitor, enhancing its ability to recognize complex scenes.

[0101] According to the above embodiments of the present invention, when the inspection area of ​​the target robot includes surrounding robots, the surrounding robots are robots in the same inspection area as the target robot. The behavior detection method of the robot may further include: acquiring surrounding environmental parameter data and surrounding image data of the surrounding robots; integrating the environmental parameter data, image data, surrounding environmental parameter data, and surrounding image data to obtain integrated information; processing the integrated information through a distributed prediction model to obtain the behavior collaborative judgment result of the target robot, wherein the distributed prediction model is pre-trained based on historical data, and the historical data includes the following information of the target robot: movement direction, speed change, and surrounding obstacle status; and obtaining the behavior data of the target robot based on the collaborative judgment result.

[0102] In this embodiment, when other robots (i.e., surrounding robots) are present in the vicinity of the target robot, the target robot's behavioral data can be determined through a collaborative judgment method. That is, when multiple inspection robots are performing tasks in the same area, an information sharing and collaboration mechanism is established between the robots. Each robot transmits the environmental information it collects and its preliminary analysis results to the surrounding robots in real time.

[0103] Specifically, a distributed collaborative prediction algorithm can be employed, where surrounding robots collaboratively determine the target robot's behavior based on information received from other robots and their own sensory data. By using a Bayesian network model and taking multi-source information as input, the probability distribution of the target robot's behavior is calculated, thus more accurately determining whether its behavior is normal. This collaborative judgment method fully utilizes collective intelligence and overcomes the limitations of individual robot perception.

[0104] For example, when multiple inspection robots enter a collaborative work area, a communication connection is first established between the robots to ensure real-time information transmission. During task execution, each robot packages and sends the environmental information it has collected and its preliminary analysis results to surrounding robots at set time intervals. Upon receiving the information, the surrounding robots integrate it with their own collected data. Then, using distributed collaborative prediction algorithms, such as Bayesian network models, they collaboratively determine the behavior of the target robot.

[0105] Optionally, the above distributed prediction model can utilize distributed collaborative prediction algorithms, such as Bayesian network models, to collaboratively determine the behavior of the target robot.

[0106] This section uses a Bayesian network model as an example. The construction of the Bayesian network model is as follows: 1) Determine variables and nodes: Based on the requirements for determining the target robot's behavior, determine the variables and nodes in the Bayesian network. Variables include the target robot's motion direction, speed changes, and the state of surrounding obstacles, with each variable corresponding to a node; 2) Define causal relationships between nodes: Through domain knowledge and data analysis, determine the causal relationships between nodes and represent them as directed edges. The target robot's speed change may be affected by its motion direction and the state of surrounding obstacles, therefore directed edges can be established between the corresponding nodes; 3) Determine the conditional probability table (CPT): For each node, determine its conditional probability table based on historical data. The conditional probability table describes the probability that the node takes different values ​​given the state of its parent node. Given the target robot's motion direction and the state of surrounding obstacles, determine the probability distribution of its speed change (i.e., the probability of the robot's speed change in each speed change interval).

[0107] For example, X1: the robot's motion state (e.g., moving forward, stopping, turning, denoted as S1); X2: the distance to obstacles in front of the robot (e.g., near, medium, far, denoted as S2); X3: the robot's speed (e.g., fast, medium, slow, denoted as S3). P(X1=S1, X2=S2, X3=S3)=P(X1=S1|X2=S2, X3=S3)P(X2=S2)P(X3=S3).

[0108] First, environmental parameter data and image data collected by surrounding robots are acquired. Next, data integration is performed: the data from the target robot and surrounding robots are combined to form a unified dataset for easier subsequent analysis. Then, a distributed prediction algorithm is used for processing. Specifically, a pre-trained distributed prediction model processes the integrated data to obtain the target robot's collaborative behavior determination result. During model training, historical collaborative task data is used for supervised learning to optimize model parameters, enabling effective prediction based on multi-robot information.

[0109] By sharing information and making collaborative judgments among robots, we can leverage swarm intelligence to improve the accuracy and reliability of behavior detection. Especially in complex scenarios, collaboration among multiple robots can overcome the limitations of a single robot's perception (potential blind spots) and improve the overall efficiency of the system.

[0110] According to the above embodiments of the present invention, after obtaining the behavior data of the target robot based on the collaborative determination result, the behavior detection method of the robot may further include: updating the behavior data using the collaborative determination result.

[0111] In this embodiment, the understanding of the target robot's behavior can be updated based on the aforementioned collaborative determination results, and the determination results can be fed back to relevant systems or operators. Here, swarm intelligence is utilized to improve the accuracy and reliability of behavior detection.

[0112] According to the above embodiments of the present invention, the robot behavior detection method further includes: acquiring environmental change information of the environment in which the target robot is located; updating the parameters of the image processing algorithm according to the environmental change information, wherein the image processing algorithm is used to process the first image data and the second image data.

[0113] To cope with the complex and ever-changing inspection environment, a dynamic environment adaptive technology is introduced. Utilizing real-time sensor data and an environmental model, the system can automatically identify dynamic changes in the environment, such as changes in light and temperature fluctuations, and adjust the parameters of the detection model accordingly. When a sudden dimming of light is detected, the parameters of the image enhancement algorithm are automatically increased to improve image clarity, ensuring that the accuracy of the detection is not affected by environmental changes.

[0114] Figure 5 This is a framework diagram of the integrated adaptation technology for the robot behavior detection method according to the present invention, such as... Figure 5 As shown, it can include dynamic environment adaptation technology, abnormal behavior early warning technology, and semantic understanding and knowledge graph technology.

[0115] Specifically, such as Figure 5As shown, for environmental change detection (dynamic environment adaptive technology), environmental change information (sensor data and environmental model) of the target robot's environment can be acquired in real time. This means recognizing dynamic environmental changes, such as changes in light intensity (below 100 lux is considered a low-light environment) and ambient temperature (above 35℃ is considered a high-temperature environment). Image processing algorithm parameters can be dynamically adjusted based on environmental change information to enhance image processing performance. For example, when an environmental change is detected, the detection model parameters are adjusted; conversely, the current detection model parameters are maintained. For instance, in a low-light environment, the brightness and contrast of the image are enhanced; in a high-temperature environment, the temperature compensation parameters of the image enhancement algorithm are adjusted.

[0116] By applying the aforementioned dynamic environment adaptation techniques, the effectiveness of the behavior detection model is ensured in constantly changing environments, improving its robustness and adaptability. Especially in environments with low light or extreme temperatures, real-time adjustment of algorithm parameters maintains high-quality image processing and guarantees the accuracy of behavior detection.

[0117] According to the above embodiments of the present invention, the behavior detection method of the robot may further include: determining the behavior pattern corresponding to the behavior data through a behavior warning model, wherein the behavior warning model is obtained by machine learning training using multiple sets of training data, historical behavior data is the input of the behavior warning model, and the historical behavior pattern corresponding to the historical behavior data is the output of the behavior warning model.

[0118] This embodiment introduces an abnormal behavior early warning technology. Based on historical data and real-time detection results, an abnormal behavior early warning model is constructed using machine learning algorithms. Through in-depth analysis of robot behavior data, potential abnormal behavior patterns are identified. Once the system detects that the current behavior matches an abnormal pattern to a set threshold, it immediately issues an early warning signal. Analysis of time-series data such as the robot's motion trajectory and operation frequency helps to detect potential faults or abnormal behaviors in advance.

[0119] Specifically, such as Figure 5As shown, abnormal behavior early warning technology can identify potential behavior patterns through a behavior early warning model, including: 1) Historical data collection: collecting historical behavior data and real-time detection results, including the target robot's motion trajectory (e.g., straight line, turning, backward), operation frequency (e.g., 5 inspection tasks per hour), and environmental parameter data related to these behaviors; 2) Model training: using machine learning algorithms, such as Long Short-Term Memory (LSTM) networks, to train the historical data and establish a behavior early warning model; and deeply mining robot behavior data (e.g., motion trajectory, operation frequency, etc.). The input to the model is real-time behavior data and environmental parameter data, and the output is the predicted behavior pattern and the probability of abnormal behavior; 3) Behavior prediction and early warning: inputting real-time behavior data into the trained model, outputting the degree of matching between the current behavior and historical behavior patterns, and early warning information on the existence of abnormal behavior. When the probability of abnormal behavior exceeds a set threshold (e.g., 0.7), the system will automatically issue an early warning signal.

[0120] In the above embodiments, potential abnormal behavior models can be identified through abnormal behavior mechanisms. Furthermore, as... Figure 5 As shown, it can also calculate the matching degree between the current behavior and the abnormal pattern, and determine whether the matching degree exceeds the threshold; if it exceeds the threshold, an early warning signal is issued, otherwise monitoring continues.

[0121] The behavioral early warning model built here enables the system to predict potential behavioral anomalies in advance, reducing the risk of failure. Through deep learning algorithms, the model can automatically learn and extract time-series features of behavior, improving the accuracy of behavior prediction and the intelligence of system decision-making.

[0122] According to the above embodiments of the present invention, the behavior detection method of the robot may further include: when the current temperature of a predetermined component of the target robot is detected to be higher than a predetermined temperature, determining the temperature information of the predetermined component through a knowledge graph, wherein the temperature information includes the current temperature and the ambient temperature of the predetermined component, and the knowledge graph is a graph pre-constructed based on the various components of the target robot and the environment in which the target robot is located; and determining the fault factors and fault handling methods of the predetermined component based on the current temperature and the ambient temperature.

[0123] This section introduces semantic understanding and knowledge graph technologies. Specifically, to enable robots to better understand the semantic information of their environment and their own behavior, semantic understanding and knowledge graph technologies are introduced. For example... Figure 5As shown, information such as devices and scenes in the environment is constructed into a knowledge graph. The robot performs semantic analysis on the collected information and matches and infers with the knowledge graph (i.e., matching and inferring the semantic analysis results with the knowledge graph); then it determines whether there is an anomaly. When the robot detects an abnormal increase in the temperature of a device (a component of the robot itself), the knowledge graph can quickly associate it with the device's relevant attributes (its own temperature and the ambient temperature), possible causes of failure, and corresponding handling measures to assist in decision-making, thus providing richer knowledge support for the robot's decision-making. If no anomaly is detected, monitoring continues.

[0124] As mentioned above, using knowledge graph technology to enhance the decision-making intelligence of robots includes: 1) Knowledge graph construction: constructing a knowledge graph from information such as equipment and scenarios in the environment, including the normal operating temperature range of the equipment (e.g., 20℃ to 40℃), the equipment's fault modes (e.g., overheating, short circuit, wear), and fault handling measures (e.g., cooling, replacing parts, repair); 2) Semantic understanding and matching: when the robot detects an abnormal temperature of a device, it performs semantic analysis and matching through the knowledge graph to quickly locate possible causes of the fault and handling measures; 3) Decision support: based on the information provided by the knowledge graph, the robot can make more reasonable decisions, such as automatically starting the cooling system when the equipment is overheating, or notifying the operator to take further action.

[0125] The application of knowledge graph technology has greatly enhanced the decision-making ability and intelligence level of robots. By storing environmental equipment information, fault modes, and handling measures in a structured manner, the robot's semantic understanding and decision-making process are more efficient, enabling it to quickly identify problems and take appropriate measures, thereby improving the system's response speed and processing efficiency.

[0126] As described above, the technical solutions provided in this embodiment of the invention mainly include: 1) Dual-channel training: Uniquely integrating environmental information collected by the robot's own sensors and cameras, as well as video image information synchronously collected by external monitors. Deep learning algorithms are used for feature extraction and feature fusion, enabling the robot to recognize complex scenes from both its own and global perspectives. 2) Collaborative judgment: In multi-robot operation scenarios, a mechanism for information sharing and collaboration among robots is constructed. A distributed collaborative prediction algorithm is used to calculate the probability distribution of the target robot's behavior based on multi-source information, compensating for the limitations of individual robot perception. 3) Dynamic environment adaptation technology: Relying on real-time sensor data and environmental models, the system automatically identifies dynamic changes in the environment such as light and temperature, and adjusts the detection model parameters accordingly to maintain detection accuracy. 4) Abnormal behavior early warning technology: Through machine learning algorithms, historical data of robot behavior is mined to construct an abnormal behavior early warning model, allowing for early detection of potential anomalies. 5) Semantic understanding and knowledge graph technology: Information such as equipment and scenes in the environment is constructed into a knowledge graph. The robot collects information through semantic analysis, matches and reasons with the knowledge graph, enriching decision-making knowledge support. The system offers the following benefits: 1) Dual-channel training effectively integrates the robot's own sensory information and external monitor information, significantly improving the robot's perception of complex environments and the accuracy of behavior detection. 2) The collaborative judgment mechanism fully leverages the cooperative advantages among multiple inspection robots, utilizing collective intelligence to enhance the reliability of robot behavior judgments and reduce misjudgments and omissions. 3) Dynamic environment adaptation technology enables the system to adapt to environmental changes in real time, ensuring the stability and accuracy of detection. 4) Abnormal behavior early warning technology detects potential problems in advance, reducing the risk of failure. 5) Semantic understanding and knowledge graph technologies provide the robot with richer knowledge support, enhancing its intelligent decision-making level.

[0127] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0129] Example 2

[0130] According to embodiments of the present invention, a robot behavior detection apparatus for implementing the above-described robot behavior detection method is also provided. Figure 6 This is a schematic diagram of a robot behavior detection device according to an embodiment of the present invention, such as... Figure 6 As shown, the robot's behavior detection device includes: a first acquisition unit 601, a processing unit 603, a first determination unit 605, and a second acquisition unit 607. The robot's behavior detection device will be described below.

[0131] The first acquisition unit 601 is used to acquire environmental parameter data and image data. The environmental parameter data is data collected by the information sensing component, and the image data includes first image data and second image data. The first image data is data collected by the first image acquisition component, and the second image data is data collected by the second image acquisition component. The information sensing component and the first image acquisition component are mounted on the target robot, and the second image acquisition component is set in the area where the target robot is located. The target robot is a robot that needs to perform behavior detection.

[0132] The processing unit 603 is used to normalize the environmental parameter data to obtain effective environmental data of the same scale, and to crop the first image data and the second image data to obtain the third image data and the fourth image data related to the target robot.

[0133] The first determining unit 605 is used to determine the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data through a feature fusion model. The convolutional neural network model of the feature fusion model is used to process the third image data and the fourth image data, and the fully connected layer of the feature fusion model is used to process the effective environmental data. The feature fusion model is a pre-trained model.

[0134] The second acquisition unit 607 is used to obtain the behavior data of the target robot based on the comprehensive feature data.

[0135] It should be noted that the first acquisition unit 601, processing unit 603, first determination unit 605 and second acquisition unit 607 mentioned above correspond to steps S202 to S208 in the above embodiments. The four units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0136] As can be seen from the above, in the scheme described in the above embodiments of the present invention, environmental parameter data and image data can be acquired using a first acquisition unit. The environmental parameter data is data collected by an information sensing component, and the image data includes first image data and second image data. The first image data is data collected by a first image acquisition component, and the second image data is data collected by a second image acquisition component. The information sensing component and the first image acquisition component are mounted on the target robot, and the second image acquisition component is located in the area where the target robot is located. The target robot is a robot that requires behavior detection. The processing unit normalizes the environmental parameter data to obtain effective environmental data of the same scale, and simultaneously crops the first and second image data to obtain third and fourth image data related to the target robot. Then, the processing unit... The first determining unit uses a feature fusion model to determine the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data. The convolutional neural network model of the feature fusion model is used to process the third and fourth image data, and the fully connected layer of the feature fusion model is used to process the effective environmental data. The feature fusion model is a pre-trained model. The second acquiring unit uses the comprehensive feature data to obtain the target robot's behavior data. This achieves the goal of fusing the robot's own sensor data, its own image data, and the image data from the external monitor. By obtaining more comprehensive information from multiple data sources, the robot's behavior detection is improved, which also improves the accuracy, robustness, and robustness of the robot's behavior detection and reduces the risk of misjudgment caused by a single data source.

[0137] Therefore, the technical solutions provided by the embodiments of the present invention solve the technical problem in the related art that in complex environments, it is difficult for a single information source to accurately determine the behavior of a robot, which easily leads to misjudgment and omission.

[0138] Optionally, the first acquisition unit includes: a trigger module, used to trigger the information sensing component to collect environmental parameter data around the target robot during the process of the target robot moving along the inspection route, and simultaneously trigger the first image acquisition component to collect first image data around the target robot and the second image acquisition component to collect second image data of the area where the target robot is located.

[0139] Optionally, the processing unit includes: a detection module for detecting anomalies in the environmental parameter data to obtain abnormal data in the environmental parameter data; a first processing module for processing the abnormal data to obtain processed environmental parameter data; and a second processing module for normalizing the processed environmental parameter data to obtain valid environmental data of the same scale.

[0140] Optionally, the processing unit includes: a first cropping module, used to crop the background area in the first image data according to the first viewing angle range of the target robot and the monitoring requirement information, to obtain third image data related to the target robot; and a second cropping module, used to crop the non-key monitoring area in the second image data that is not related to the target robot according to the monitoring requirement information and the second viewing angle range of the second image acquisition component, to obtain fourth image data related to the target robot.

[0141] Optionally, the robot's behavior detection device further includes: a first calibration unit, used to calibrate the information sensing component using historical temperature data and historical humidity data before acquiring environmental parameter data and image data, wherein the historical temperature data and historical humidity data are data collected by the information sensing component within a historical time period; and a second calibration unit, used to calibrate the first image acquisition component according to an intrinsic parameter matrix and an extrinsic parameter matrix, wherein the intrinsic parameter matrix and extrinsic parameter matrix are calculated based on sample images acquired by the first image acquisition component from different angles.

[0142] Optionally, the robot's behavior detection device further includes: a first initialization unit, used to initialize the target robot's operating system and data acquisition program before acquiring environmental parameter data and image data, in order to set the data acquisition frequency and data storage path of the information sensing component and the first image acquisition component; and a second initialization unit, used to initialize the second image acquisition component, in order to set the image acquisition parameters and image acquisition range of the second image acquisition component, wherein the image acquisition range is associated with the map corresponding to the monitoring area of ​​the second image acquisition component.

[0143] Optionally, the first determining unit includes: an extraction module, used to extract features from the third image data and the fourth image data using a convolutional neural network model in the feature fusion model to obtain the first image features and the second image features of the target robot, and to extract features from the effective environmental data using a fully connected layer to obtain environmental parameter features; and a stitching module, used to stitch the first image features, the second image features and the environmental parameter features using a fully connected layer to obtain comprehensive feature data.

[0144] Optionally, the robot's behavior detection device further includes: a third acquisition unit, used to acquire surrounding environmental parameter data and surrounding image data of the surrounding robots when the target robot's inspection area includes surrounding robots (the surrounding robots are robots in the same inspection area as the target robot); an integration unit, used to integrate the environmental parameter data, image data, surrounding environmental parameter data, and surrounding image data to obtain integrated information; a fourth acquisition unit, used to process the integrated information through a distributed prediction model to obtain the target robot's behavior collaborative judgment result, wherein the distributed prediction model is pre-trained based on historical data, and the historical data includes the following information of the target robot: movement direction, speed change, and surrounding obstacle status; and a fifth acquisition unit, used to obtain the target robot's behavior data based on the collaborative judgment result.

[0145] Optionally, the robot's behavior detection device further includes: a first update unit, used to update the behavior data using the collaborative determination result after obtaining the target robot's behavior data based on the collaborative determination result.

[0146] Optionally, the robot's behavior detection device further includes: a sixth acquisition unit for acquiring environmental change information of the environment in which the target robot is located; and a second update unit for updating the parameters of the image processing algorithm based on the environmental change information, wherein the image processing algorithm is used to process the first image data and the second image data.

[0147] Optionally, the robot's behavior detection device further includes: a second determining unit, used to determine the behavior pattern corresponding to the behavior data through a behavior warning model, wherein the behavior warning model is obtained by machine learning training using multiple sets of training data, historical behavior data is the input of the behavior warning model, and the historical behavior pattern corresponding to the historical behavior data is the output of the behavior warning model.

[0148] Optionally, the robot's behavior detection device further includes: a third determining unit, used to determine the temperature information of the predetermined component through a knowledge graph when the current temperature of the predetermined component of the target robot is detected to be higher than the predetermined temperature, wherein the temperature information includes the current temperature and the ambient temperature of the predetermined component, and the knowledge graph is a graph pre-constructed based on the various components of the target robot and the environment in which the target robot is located; and a fourth determining unit, used to determine the fault factors and fault handling methods of the predetermined component based on the current temperature and the ambient temperature.

[0149] According to another aspect of the present invention, a robot behavior detection system is also provided, which uses any of the robot behavior detection methods described above.

[0150] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes the robot behavior detection method described above.

[0151] According to another aspect of the present invention, a computer program product is also provided, including computer instructions, which, when executed by a processor, perform a robot behavior detection method according to any one of the above.

[0152] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes the robot behavior detection method of any of the above.

[0153] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any communication device in a group of communication devices.

[0154] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring environmental parameter data and image data, wherein the environmental parameter data is data collected by an information sensing component, and the image data includes first image data and second image data, wherein the first image data is data collected by a first image acquisition component, and the second image data is data collected by a second image acquisition component, the information sensing component and the first image acquisition component are mounted on a target robot, the second image acquisition component is located in the area where the target robot is located, and the target robot is a robot for which behavior detection is required; normalizing the environmental parameter data to obtain effective environmental data of the same scale, and simultaneously cropping the first image data and the second image data to obtain third image data and fourth image data related to the target robot; determining the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data through a feature fusion model, wherein the convolutional neural network model of the feature fusion model is used to process the third image data and the fourth image data, the fully connected layer of the feature fusion model is used to process the effective environmental data, and the feature fusion model is a pre-trained model; and obtaining the behavior data of the target robot based on the comprehensive feature data.

[0155] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: during the process of the target robot moving along the inspection route, the information sensing component is triggered to collect environmental parameter data around the target robot, and the first image acquisition component is triggered to collect first image data around the target robot, and the second image acquisition component is triggered to collect second image data of the area where the target robot is located.

[0156] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: performing anomaly detection on environmental parameter data to obtain abnormal data in the environmental parameter data; processing the abnormal data to obtain processed environmental parameter data; and normalizing the processed environmental parameter data to obtain valid environmental data of the same scale.

[0157] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: cropping the background area in the first image data according to the first view range of the target robot and the monitoring requirement information to obtain third image data related to the target robot; cropping the non-key monitoring area unrelated to the target robot in the second image data according to the monitoring requirement information and the second view range of the second image acquisition component to obtain fourth image data related to the target robot.

[0158] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: calibrating the information sensing component using historical temperature data and historical humidity data before acquiring environmental parameter data and image data, wherein the historical temperature data and historical humidity data are data collected by the information sensing component within a historical time period; calibrating the first image acquisition component according to an intrinsic parameter matrix and an extrinsic parameter matrix, wherein the intrinsic parameter matrix and extrinsic parameter matrix are calculated based on sample images acquired by the first image acquisition component from different angles.

[0159] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: before acquiring environmental parameter data and image data, the operating system and data acquisition program of the target robot are initialized to set the data acquisition frequency and data storage path of the information sensing component and the first image acquisition component; the second image acquisition component is initialized to set the image acquisition parameters and image acquisition range of the second image acquisition component, wherein the image acquisition range is associated with the map corresponding to the monitoring area of ​​the second image acquisition component.

[0160] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in the feature fusion model, a convolutional neural network model is used to extract features from the third image data and the fourth image data to obtain the first image features and the second image features of the target robot, and a fully connected layer is used to extract features from the effective environmental data to obtain environmental parameter features; the fully connected layer is used to concatenate the first image features, the second image features, and the environmental parameter features to obtain comprehensive feature data.

[0161] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: when the inspection area of ​​the target robot includes surrounding robots, the surrounding robots are robots in the same inspection area as the target robot; acquiring the surrounding environmental parameter data and surrounding image data of the surrounding robots; integrating the environmental parameter data, image data, surrounding environmental parameter data, and surrounding image data to obtain integrated information; processing the integrated information through a distributed prediction model to obtain the behavior collaborative judgment result of the target robot, wherein the distributed prediction model is pre-trained based on historical data, and the historical data includes the following information of the target robot: movement direction, speed change, and surrounding obstacle status; obtaining the behavior data of the target robot based on the collaborative judgment result.

[0162] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: after obtaining the behavior data of the target robot based on the cooperative determination result, updating the behavior data using the cooperative determination result.

[0163] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining environmental change information of the environment in which the target robot is located; updating the parameters of the image processing algorithm according to the environmental change information, wherein the image processing algorithm is used to process the first image data and the second image data.

[0164] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining the behavior pattern corresponding to the behavior data through a behavior warning model, wherein the behavior warning model is obtained by machine learning training using multiple sets of training data, historical behavior data is the input of the behavior warning model, and the historical behavior pattern corresponding to the historical behavior data is the output of the behavior warning model.

[0165] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: when the current temperature of a predetermined component of the target robot is detected to be higher than a predetermined temperature, the temperature information of the predetermined component is determined through a knowledge graph, wherein the temperature information includes the current temperature and the ambient temperature of the predetermined component, and the knowledge graph is a graph pre-constructed based on the various components of the target robot and the environment in which the target robot is located; and the fault factors and fault handling methods of the predetermined component are determined based on the current temperature and the ambient temperature.

[0166] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0167] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0172] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting the behavior of a robot, characterized in that, include: The system acquires environmental parameter data and image data. The environmental parameter data is obtained by triggering an information sensing component to collect data around the target robot as it moves along the inspection route. The image data includes first image data and second image data. The first image data is collected by a first image acquisition component to collect data around the target robot as it moves along the inspection route. The second image data is collected by a second image acquisition component to collect data in the area where the target robot is located as it moves along the inspection route. The information sensing component and the first image acquisition component are mounted on the target robot, and the second image acquisition component is located in the area where the target robot is located. The target robot is a robot that needs to perform behavior detection. The environmental parameter data is normalized to obtain effective environmental data of the same scale. At the same time, the first image data and the second image data are cropped to obtain third image data and fourth image data related to the target robot. A feature fusion model is used to determine the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data. In this model, a convolutional neural network model is used to extract features from the third and fourth image data to obtain first and second image features of the target robot, and a fully connected layer is used to extract features from the effective environmental data to obtain environmental parameter features. The fully connected layer then concatenates the first image features, the second image features, and the environmental parameter features to obtain the comprehensive feature data. The feature fusion model is a pre-trained model. The behavioral data of the target robot are obtained based on the comprehensive feature data.

2. The robot behavior detection method according to claim 1, characterized in that, The environmental parameter data is normalized to obtain effective environmental data with the same scale, including: Anomaly detection is performed on the environmental parameter data to obtain abnormal data in the environmental parameter data; The abnormal data is processed to obtain the processed environmental parameter data; The processed environmental parameter data is normalized to obtain valid environmental data of the same scale.

3. The robot behavior detection method according to claim 1, characterized in that, The first image data and the second image data are cropped to obtain the third image data and the fourth image data related to the target robot, including: Based on the first viewpoint range of the target robot and the monitoring requirements, the background area in the first image data is cropped to obtain the third image data related to the target robot; Based on the monitoring requirements and the second viewpoint range of the second image acquisition component, the non-key monitoring areas unrelated to the target robot in the second image data are cropped to obtain the fourth image data related to the target robot.

4. The robot behavior detection method according to claim 1, characterized in that, Before acquiring environmental parameter data and image data, the process also includes: The information sensing component is calibrated using historical temperature data and historical humidity data, wherein the historical temperature data and historical humidity data are data collected by the information sensing component within a historical time period; The first image acquisition component is calibrated based on the intrinsic and extrinsic parameter matrices, wherein the intrinsic and extrinsic parameter matrices are calculated based on sample images acquired by the first image acquisition component from different angles.

5. The robot behavior detection method according to claim 1, characterized in that, Before acquiring environmental parameter data and image data, the process also includes: The operating system and data acquisition program of the target robot are initialized to set the data acquisition frequency and data storage path of the information sensing component and the first image acquisition component; The second image acquisition component is initialized to set its image acquisition parameters and image acquisition range, wherein the image acquisition range is associated with the map corresponding to the monitoring area of ​​the second image acquisition component.

6. The robot behavior detection method according to claim 1, characterized in that, When the inspection area of ​​the target robot includes surrounding robots, the surrounding robots are robots located in the same inspection area as the target robot, and also include: Acquire the surrounding environmental parameter data and surrounding image data of the surrounding robots; The environmental parameter data, the image data, the surrounding environmental parameter data, and the surrounding image data are integrated to obtain integrated information; The integrated information is processed by a distributed prediction model to obtain the behavior coordination judgment result of the target robot. The distributed prediction model is pre-trained based on historical data, which includes the following information of the target robot: movement direction, speed change, and surrounding obstacle status. The behavioral data of the target robot are obtained based on the collaborative determination result.

7. The robot behavior detection method according to claim 6, characterized in that, After obtaining the behavioral data of the target robot based on the collaborative determination result, the method further includes: The behavioral data is updated using the collaborative determination results.

8. The robot behavior detection method according to claim 1, characterized in that, Also includes: Obtain environmental change information about the environment in which the target robot is located; The parameters of the image processing algorithm are updated based on the environmental change information, wherein the image processing algorithm is used to process the first image data and the second image data.

9. The robot behavior detection method according to claim 1, characterized in that, Also includes: The behavioral pattern corresponding to the behavioral data is determined by a behavioral early warning model, wherein the behavioral early warning model is trained by machine learning using multiple sets of training data, historical behavioral data is the input of the behavioral early warning model, and the historical behavioral pattern corresponding to the historical behavioral data is the output of the behavioral early warning model.

10. The robot behavior detection method according to any one of claims 1 to 9, characterized in that, Also includes: When the current temperature of a predetermined component of the target robot is detected to be higher than a predetermined temperature, the temperature information of the predetermined component is determined through a knowledge graph. The temperature information includes the current temperature and the ambient temperature of the predetermined component. The knowledge graph is a pre-constructed graph based on the various components of the target robot and the environment in which the target robot is located. The fault factors and fault handling methods of the predetermined component are determined based on the current temperature and the ambient temperature.

11. A robot behavior detection device, characterized in that, include: The first acquisition unit is used to acquire environmental parameter data and image data. The environmental parameter data is obtained by triggering an information sensing component to collect data around the target robot during the movement of the target robot along the inspection route. The image data includes first image data and second image data. The first image data is obtained by triggering a first image acquisition component to collect data around the target robot during the movement of the target robot along the inspection route. The second image data is obtained by triggering a second image acquisition component to collect data in the area where the target robot is located during the movement of the target robot along the inspection route. The information sensing component and the first image acquisition component are mounted on the target robot, and the second image acquisition component is located in the area where the target robot is located. The target robot is a robot that needs to perform behavior detection. The processing unit is used to normalize the environmental parameter data to obtain effective environmental data of the same scale, and simultaneously to crop the first image data and the second image data to obtain third image data and fourth image data related to the target robot. The first determining unit is configured to determine the comprehensive feature data of the target robot corresponding to the effective environmental data, the third image data, and the fourth image data through a feature fusion model. In the feature fusion model, a convolutional neural network model is used to extract features from the third and fourth image data to obtain first and second image features of the target robot, and a fully connected layer is used to extract features from the effective environmental data to obtain environmental parameter features. The fully connected layer is then used to concatenate the first image features, the second image features, and the environmental parameter features to obtain the comprehensive feature data. The feature fusion model is a pre-trained model. The second acquisition unit is used to obtain the behavior data of the target robot based on the comprehensive feature data.

12. A robot behavior detection system, characterized in that, The robot behavior detection system uses the robot behavior detection method described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes the robot behavior detection method according to any one of claims 1 to 10.

14. A processor, characterized in that, The processor is used to run a program, wherein the program executes the robot behavior detection method according to any one of claims 1 to 10.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, the robot behavior detection method according to any one of claims 1 to 10 is performed.

Images