Abnormality detection method, device and equipment applied to inspection robot and medium

Abstract

The application discloses an abnormality detection method, device and equipment applied to a patrol robot, and a medium, relates to the cross technical field of robot autonomous navigation and computer vision, and the method comprises the following steps: collecting a video stream on a patrol path, extracting a key frame, and processing an input tensor; obtaining a feature map based on the input tensor through an image encoder of a pre-trained visual language large model, processing a visual feature vector, calculating the cosine similarity of the visual feature vector and a normal feature prototype vector, determining whether an abnormal target exists, processing an abnormal heat map based on the cosine similarity and the feature map, and determining the position coordinates of each abnormal target. The application converts the abnormality detection into a similarity measurement problem of the visual feature vector and the normal feature prototype vector, avoids the dependence on negative sample training, makes the model unnecessary to learn all possible abnormal forms in advance, and solves the missing detection problem caused by the scarcity of negative samples.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of robot autonomous navigation and computer vision, and in particular to an anomaly detection method, device, equipment and medium for inspection robots. Background Technology

[0002] Inspection robots used in industrial production, public safety, and household services need to be able to "perceive and understand" the environmental conditions to promptly detect anomalies (such as cluttered corridors, damaged equipment, and leaking liquids on the floor). This is crucial for ensuring production safety and maintaining environmental order. Related technologies typically involve inspection robots collecting image data in real time and using machine vision-based anomaly detection techniques to achieve anomaly detection.

[0003] However, the relevant technologies still face the following application challenges: First, existing mainstream anomaly detection algorithms (such as supervised learning-based classification models) typically require a large number of "positive samples" (normal images) and "negative samples" (abnormal images) for training. While data from normal environments is readily available, anomalies are often sporadic, sparse, and varied in form (i.e., long-tailed distribution), making it difficult to exhaustively enumerate all possible anomaly samples for model training. Once an anomaly type not included in the training set is encountered, the recognition rate of traditional models drops significantly, leading to missed detections. On the other hand, while traditional unsupervised anomaly detection methods do not require negative samples, they need to "memorize" the pixel distribution of normal samples. This approach can only identify pixel-level statistical biases and lacks an understanding of scene semantics. When the environment changes, such as changes in lighting, minor adjustments in perspective, or the appearance of dynamic objects unrelated to the task (such as people walking normally in a corridor), these methods are prone to false alarms, requiring data collection and model retraining, and they cannot understand the definition of "abnormality" in different scenarios. On the other hand, existing inspection robots are unable to intuitively present the source of the anomaly to users. At the same time, inspection robots have difficulty understanding human natural language commands. For example, users cannot simply tell the robot "go check if the ground is clean." This semantic gap limits the flexibility and ease of use of robots in complex human-machine collaboration scenarios.

[0004] In summary, how to break free from the dependence on negative sample training, utilize general semantic knowledge to achieve anomaly detection in unknown and open scenarios, and possess good environmental adaptability and interpretability are key issues that current mobile robot inspection technology urgently needs to address. Summary of the Invention

[0005] This application provides an anomaly detection method, apparatus, equipment, and medium for inspection robots, to address the shortcomings of the aforementioned related technologies. The technical solution is as follows:

[0006] Firstly, this application provides an anomaly detection method for inspection robots, comprising:

[0007] The inspection robot's image acquisition device collects video streams along the inspection path;

[0008] Based on the video stream, keyframes are sampled to extract multiple keyframes, and each keyframe is processed to obtain an input tensor.

[0009] The image encoder of the pre-trained visual language large model extracts the image features of the input tensor to obtain a feature map, and the visual feature vector is obtained by processing the feature map.

[0010] Calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and determine whether there is an abnormal target based on the cosine similarity.

[0011] If the existence of abnormal targets is determined, an abnormal heat map is obtained based on the cosine similarity and the feature map to describe the distribution of abnormal targets;

[0012] The location coordinates of each abnormal target are determined based on the aforementioned anomaly heatmap.

[0013] In one alternative to the first aspect, the process of obtaining the input tensor based on each keyframe includes:

[0014] Scale the size of the keyframe to a preset input size;

[0015] Center-crop the scaled keyframes to remove edge distortion;

[0016] The keyframes after center clipping are converted into tensors and then normalized.

[0017] Based on the pre-trained visual language large model, the mean and standard deviation of the pixel values ​​are used to standardize the normalized tensor, generating an input tensor of a preset dimension.

[0018] In one alternative embodiment of the first aspect, the steps for constructing the normal feature prototype vector include:

[0019] Multiple corpora describing the scene where the inspection robot is located as being in a normal state are obtained, and prompt words are constructed based on each corpus; wherein, the normal state means that there are no abnormal targets in the scene where the inspection robot is located;

[0020] Each prompt word is input into the visual language big model, the prompt word is converted into a word sequence, and the text features of the word sequence are extracted by the text encoder of the visual language big model to obtain the corresponding text feature vectors.

[0021] The normal feature prototype vector is obtained by fusing features based on all text feature vectors.

[0022] In one alternative to the first aspect, calculating the cosine similarity between the visual feature vector and a pre-constructed normal feature prototype vector, and determining whether an anomalous target exists based on the cosine similarity, includes:

[0023] The dot product operation is performed on the visual feature vector and the normal feature prototype vector to obtain the cosine similarity between the two vectors. The cosine similarity represents the degree of matching between the keyframe corresponding to the visual feature vector and the normal state.

[0024] Compare the cosine similarity of each keyframe with the cosine similarity threshold. If the cosine similarity of the current keyframe and the consecutive preset number of keyframes before the current keyframe are all less than the cosine similarity threshold, then it is determined that there is an abnormal target in the current keyframe.

[0025] In one alternative to the first aspect, the method further includes, prior to deploying the inspection robot:

[0026] The inspection robot is deployed in a scene where there are no abnormal targets and the scene is in a normal state. The inspection robot collects normal environmental video streams of the scene in a normal state.

[0027] Calculate the cosine similarity between the visual feature vector of each image frame in the normal environment video stream and the normal feature prototype vector, and calculate the mean and standard deviation of the cosine similarity.

[0028] The cosine similarity threshold is calculated based on the mean and standard deviation of the cosine similarity.

[0029] In one alternative embodiment of the first aspect, obtaining the anomaly heatmap for describing the distribution of anomalous targets based on the cosine similarity and the feature map processing includes:

[0030] The objective function is constructed based on the difference between the cosine similarity of each keyframe and 1.

[0031] Obtain the pixel value of each pixel position in each feature channel in the feature map, and calculate the gradient of the objective function with respect to each pixel value based on the backpropagation algorithm;

[0032] Global average pooling is performed on each gradient to calculate the weights corresponding to each feature channel of the feature map.

[0033] Based on the weights, each feature channel of the feature map is linearly weighted and superimposed. The weighted and superimposed feature map is then processed by an activation function to filter out negative values, resulting in a heat map value at each pixel location. The abnormal heat map is then constructed based on the heat map value at each pixel location.

[0034] In one alternative embodiment of the first aspect, determining the location coordinates of each anomalous target based on the anomalous heatmap includes:

[0035] The abnormal heat map is binarized, and heat values ​​greater than the heat value threshold are set to 1, while heat values ​​less than or equal to the heat value threshold are set to 0.

[0036] Determine the connected regions composed of pixels with a thermal value of 1 in the abnormal heat map, determine the centroid of each connected region, and calculate the centroid coordinates.

[0037] The depth value of the image acquisition device at the centroid coordinates is obtained, and the spatial coordinates of the centroid relative to the inspection robot are calculated based on the camera intrinsic parameter matrix of the image acquisition device and the depth value, so as to obtain the position coordinates of the abnormal target at the centroid position.

[0038] Secondly, this application also provides an anomaly detection device for inspection robots, comprising:

[0039] The image acquisition unit is used to acquire video streams along the inspection path through the image acquisition device of the inspection robot.

[0040] The image processing unit is used to perform keyframe sampling based on the video stream, extract multiple keyframes, and process each keyframe to obtain an input tensor.

[0041] The feature extraction unit is used to extract image features from the input tensor through the image encoder of the pre-trained visual language large model to obtain a feature map, and to process the feature map to obtain a visual feature vector.

[0042] An anomaly detection unit is used to calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and to determine whether there is an abnormal target based on the cosine similarity.

[0043] If the presence of an abnormal target is determined, the anomaly detection unit is also used to obtain an anomaly heatmap describing the distribution of the abnormal target based on the cosine similarity and the feature map processing.

[0044] The anomaly detection unit is also used to determine the location coordinates of each anomaly target based on the anomaly heatmap.

[0045] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method provided by the first aspect of this application or any implementation thereof.

[0046] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of this application or any implementation thereof.

[0047] The beneficial effects of the technical solution provided in this application include at least the following:

[0048] The anomaly detection method for inspection robots provided in this application extracts visual feature vectors by directly calling the image encoder of a pre-trained visual language large model, and transforms anomaly detection into a similarity measurement problem between visual feature vectors and normal feature prototype vectors. This avoids the dependence on training with negative samples (abnormal images), so that the model can effectively identify unknown anomaly types outside the training set without having to learn all possible anomaly forms in advance, thus solving the problem of missed detection caused by the scarcity of negative samples.

[0049] Secondly, since the extracted visual feature vectors contain rich semantic information, the detection process is based on high-level scene understanding rather than pixel-level statistics, which significantly enhances the robustness to dynamic changes in the environment. It can effectively distinguish between real anomalies and interference such as changes in lighting, differences in viewpoint, or irrelevant dynamic objects, greatly reducing the false alarm rate caused by the lack of semantic understanding in traditional unsupervised methods, and improving adaptability and reliability in complex environments.

[0050] Finally, by generating an anomaly heatmap that intuitively displays the area where the abnormal target is located and outputting precise location coordinates, a clear visual interpretation of the detection results is provided, enabling users to quickly locate and understand the source of the anomaly, and providing feasible technical support for intelligent inspection. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating an anomaly detection method for an inspection robot provided in an embodiment of this application;

[0053] Figure 2This is a schematic diagram of the structure of an anomaly detection device applied to an inspection robot, provided in an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0057] It should be noted that the terms "first" and "second" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than those described or illustrated herein.

[0058] The present application will now be described in detail with reference to specific embodiments.

[0059] Next, combine Figure 1 This application describes an anomaly detection method for inspection robots, provided by an embodiment of this application. For details, please refer to... Figure 1 , Figure 1 This illustration shows a flowchart of an anomaly detection method for an inspection robot provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0060] S101, the inspection robot acquires video streams along the inspection path using its image acquisition device;

[0061] S102, Keyframe sampling is performed based on the video stream to extract multiple keyframes, and an input tensor is obtained by processing each keyframe.

[0062] S103, the image encoder of the pre-trained visual language large model extracts the image features of the input tensor to obtain a feature map, and the visual feature vector is obtained by processing the feature map;

[0063] S104, calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and determine whether there is an abnormal target based on the cosine similarity.

[0064] S105, if it is determined that there are abnormal targets, an abnormal heat map is obtained based on the cosine similarity and the feature map to describe the distribution of abnormal targets.

[0065] S106, Determine the location coordinates of each abnormal target based on the abnormal heat map.

[0066] In some embodiments, the inspection robot in S101 can be the Unitree B1 industrial-grade quadruped robot platform. The image acquisition device carried by the inspection robot can include, but is not limited to, various lenses such as RGB cameras and depth cameras. For example, an Intel RealSense D435i depth camera can be selected. This application embodiment does not limit this.

[0067] Specifically, the video stream resolution and frame rate of the image acquisition device can be preset, for example, the video stream resolution can be set to 640x480 and the frame rate to 30fps.

[0068] Specifically, the inspection robot can respond to the inspection command issued by the user and control the image acquisition device on the inspection robot to collect environmental video streams on the current inspection path in real time.

[0069] In some embodiments, in S102, in order to achieve a balance between computational accuracy and system real-time performance, the system does not process all image frames in the video stream frame by frame, but instead adopts an adaptive keyframe extraction strategy based on motion state, that is, extracts keyframes from the video stream for processing according to the robot's motion state.

[0070] Specifically, keyframes can be captured from the video stream based on a preset sampling rate. The sampling rate can be reduced when the inspection robot moves quickly to reduce interference from blurry frames, and the sampling rate can be increased when the inspection robot moves slowly or is stationary to obtain clear details.

[0071] For example, the speedometer data of the robot chassis can be read. If the speed is greater than 0.5 m / s, the image is read once every 5 frames as a keyframe; if the speed is less than 0.5 m / s, the image is read frame by frame and each frame is used as a keyframe to reduce the impact of motion blur on feature extraction.

[0072] In some embodiments, the process of obtaining the input tensor based on each keyframe in S102 includes:

[0073] S1021, Scale the size of the keyframe to a preset input size;

[0074] Specifically, the preset input size refers to the standard input size of the image encoder in Contrastive Language-Image Pre-training (CLIP). For example, the OpenCV resize function can be called to scale the captured image to 224. 224 pixels.

[0075] S1022, perform center cropping on the scaled keyframes to remove edge distortion.

[0076] For example, the image center 224 of a keyframe can be captured. Area 224.

[0077] S1023, convert the keyframe after center cropping into a PyTorch tensor and perform normalization processing, dividing the pixel value by 255 to normalize it to the [0, 1] interval.

[0078] S1024, Based on the mean and standard deviation of the pixel values ​​preset by the pre-trained visual language large model, the normalized tensor is standardized to generate an input tensor of a preset dimension.

[0079] For example, by standardizing using preset mean (0.4814, 0.4578, 0.4082) and standard deviation (0.2686, 0.2613, 0.2757), an input tensor with preset dimensions [1, 3, 224, 224] is obtained.

[0080] Specifically, the visual language big model provided in this application includes a Vision Transformer image encoder and a Text Transformer text encoder.

[0081] For example, the specific model selected for the large visual language model could be ViT-B / 32.

[0082] Furthermore, the image encoder based on the large visual language model performs step S103, feeding the input tensor into the image encoder and extracting image features through the image encoder to obtain a feature map;

[0083] Further, L2 normalization is performed on the feature map to obtain the visual feature vector. The dimension of the visual feature vector can be 512.

[0084] In some embodiments, normal feature prototype vectors can be constructed by inputting a large corpus into a large visual language model. The specific construction steps include:

[0085] S201, Obtain multiple corpora describing the scene where the inspection robot is located as being in a normal state, and construct prompt words based on each corpus;

[0086] Specifically, each corpus consists of natural language text describing the scene in which the inspection robot is located as being in a normal state, wherein the normal state means that there are no abnormal targets in the scene in which the inspection robot is located.

[0087] The purpose of using multiple corpora instead of a single corpus is to cover different aspects of normal scenarios and improve the system's generalization ability.

[0088] For example, the prompt words may include:

[0089] "A photo of a clean server room floor"

[0090] "A tidy corridor with no obstacles"

[0091] "empty passage"

[0092] "Standard industrial floor".

[0093] S202, each prompt word is input into the visual language big model, the prompt word is converted into a word sequence (i.e., a token sequence), and the text features of the word sequence are extracted by the text encoder of the visual language big model to obtain the corresponding text feature vectors.

[0094] S203, perform feature fusion based on all text feature vectors to obtain the normal feature prototype vector.

[0095] Specifically, feature fusion based on all text feature vectors can be understood as calculating the arithmetic mean of all text feature vectors. The result of feature fusion can then be further normalized to obtain the normal feature prototype vector. .

[0096] In some embodiments, S104 specifically includes:

[0097] S1041, perform a dot product operation based on the visual feature vector and the normal feature prototype vector to obtain the cosine similarity between the two vectors.

[0098] Specifically, cosine similarity can be denoted as S. Cosine similarity is used to intuitively represent the degree of matching between the keyframe corresponding to the visual feature vector and the scene in the normal state. It can be understood as the semantic similarity between the visual feature vector and the normal feature prototype vector. The higher the cosine similarity, the lower the probability of an abnormal target in the keyframe, and the more the keyframe conforms to the description of "neat and clean". Otherwise, the higher the probability of an abnormal target in the keyframe.

[0099] S1042, compare the cosine similarity of each key frame with the cosine similarity threshold. If the cosine similarity of the current key frame and the consecutive preset number of key frames before the current key frame are all less than the cosine similarity threshold, then it is determined that there is an abnormal target in the current key frame.

[0100] Specifically, if This indicates that the current keyframe corresponds to a scene in a normal state;

[0101] like This retrieves the keyframe sequence of the current keyframe and the previous preset number of keyframes. For example, if the preset number is set to 5, the keyframe sequence will be retrieved in chronological order. ,in For the current keyframe, For keyframes preceding the current keyframe, if the cosine similarity of 5 consecutive frames is less than the cosine similarity threshold, then the current keyframe is considered to be... An abnormal target exists.

[0102] In some embodiments, before deploying the inspection robot, a cosine similarity threshold can be adaptively calculated from video data collected by the inspection robot under normal conditions, including:

[0103] S301, The inspection robot is deployed in a scene where there are no abnormal targets and the scene is in a normal state, and the inspection robot collects a normal environmental video stream of the scene in a normal state.

[0104] S302, calculate the cosine similarity between the visual feature vector of each image frame in the normal environment video stream and the normal feature prototype vector, and calculate the mean of the cosine similarity. and standard deviation ;

[0105] S303, the cosine similarity threshold is calculated based on the mean and standard deviation of the cosine similarity. Apply the formula:

[0106] ;

[0107] For example, mean The standard deviation is 0.85. The cosine similarity threshold is calculated to be 0.03. It is 0.76.

[0108] In some embodiments, if an anomalous target is determined to exist, step S105 is performed to precisely locate the image region causing the reduced cosine similarity, specifically including:

[0109] S1051, based on the cosine similarity of each keyframe The objective function is constructed from the difference between 1 and 1. , represented as:

[0110] ;

[0111] Wherein, objective function Cosine similarity The complement of.

[0112] S1052, obtain the pixel value of each pixel position in each feature channel in the feature map, and calculate the gradient of the objective function with respect to each pixel value based on the backpropagation algorithm.

[0113] Specifically, the feature map can be represented as A, and the feature map A's... Each feature channel is represented as At each pixel location in feature map A The pixel value on each feature channel k is represented as The gradient obtained is .

[0114] S1053, Perform global average pooling on each gradient to calculate the weights corresponding to each feature channel of the feature map, applying the formula:

[0115]

[0116] in, The weight is represented by , and N represents the total number of pixels in the feature map.

[0117] S1054, based on the weights For each feature channel of the feature map Linear weighted summation is performed, and the resulting feature map is processed using the ReLU activation function to filter out negative values ​​(i.e., retain regions that positively contribute to anomaly detection), obtaining the heatmap value at each pixel location. The anomaly heatmap is then constructed based on the heatmap values ​​at each pixel location. , represented as:

[0118]

[0119] Furthermore, it is also possible to analyze abnormal heatmaps. Post-processing is performed, with the input size as... Keyframes, spatial dimensions are Taking a feature map with 512 channels as an example, the generated... The size of the anomaly heatmap was upsampled using bilinear interpolation. The original input size is used, and OpenCV's Jet color map is applied to overlay each heat value onto the original grayscale image to form an intuitive anomaly indicator.

[0120] Understandably, the higher the heat value, the higher the probability that there is an abnormal target (obstacle) at the corresponding location. A mapping relationship between heat values ​​and RGB colors can be constructed, and the distribution of abnormal targets can be intuitively reflected on the abnormal heat map through color.

[0121] Furthermore, the steps in S106 include:

[0122] S1061, the abnormal heat map is binarized, and heat values ​​greater than the heat value threshold are set to 1, and heat values ​​less than or equal to the heat value threshold are set to 0.

[0123] S1062, determine the connected components formed by pixels with a thermal value of 1 in the abnormal heatmap, determine the centroid of each connected component and calculate the centroid coordinates. ;

[0124] A connected component is an image region consisting of pixels with the same pixel value and adjacent positions.

[0125] S1063, obtain the depth value Z of the image acquisition device at the centroid coordinates, calculate the spatial coordinates of the centroid relative to the inspection robot based on the camera intrinsic parameter matrix of the image acquisition device and the depth value, and obtain the position coordinates of the abnormal target at the centroid position.

[0126] Specifically, the camera intrinsic parameter matrix includes focal length. He Guangxin The spatial coordinates of the centroid relative to the inspection robot are calculated based on the camera intrinsic parameter matrix, using the following formula:

[0127] ;

[0128] ;

[0129] This allows us to obtain the spatial coordinates of the centroid relative to the inspection robot. That is, the location coordinates of the abnormal target.

[0130] In some embodiments, after obtaining the location coordinates of the abnormal target, a ROS (Robot Operating System) message can be constructed by the program on the inspection robot. This message contains the abnormal target type label "UnknownAnomaly", the image data and spatial coordinates of the abnormal target, and is sent to the user's terminal for the user to view.

[0131] In some embodiments, the navigation planning module of the inspection robot can mark the corresponding location as an obstacle in the local cost map of the robot navigation based on the spatial coordinates of the abnormal target, and avoid the area corresponding to the obstacle during the subsequent inspection path planning process. If the area corresponding to the obstacle completely blocks the inspection path, the inspection robot stops moving and sends an alarm information containing the bounding box of the abnormal target and its location coordinates to the user terminal via the network for manual confirmation and processing.

[0132] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0133] Please see below. Figure 2 The following is a schematic diagram of an anomaly detection device for an inspection robot, provided as an exemplary embodiment of this application. The device includes:

[0134] The image acquisition unit is used to acquire video streams along the inspection path through the image acquisition device of the inspection robot.

[0135] The image processing unit is used to perform keyframe sampling based on the video stream, extract multiple keyframes, and process each keyframe to obtain an input tensor.

[0136] The feature extraction unit is used to extract image features from the input tensor through the image encoder of the pre-trained visual language large model to obtain a feature map, and to process the feature map to obtain a visual feature vector.

[0137] An anomaly detection unit is used to calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and to determine whether there is an abnormal target based on the cosine similarity.

[0138] If the presence of an abnormal target is determined, the anomaly detection unit is also used to obtain an anomaly heatmap describing the distribution of the abnormal target based on the cosine similarity and the feature map processing.

[0139] The anomaly detection unit is also used to determine the location coordinates of each anomaly target based on the anomaly heatmap.

[0140] It should be noted that the apparatus provided in the above embodiments, when executing an anomaly detection method applied to an inspection robot, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process is detailed in the method embodiments, which will not be repeated here.

[0141] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0142] Please see Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0143] like Figure 3 As shown, the electronic device includes a processor and a memory.

[0144] In this embodiment, the processor is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array).

[0145] A processor can also include a main processor and a coprocessor. The main processor is used to process data in the wake-up state and is also called the CPU (Central Processing Unit). The coprocessor is a low-power processor used to process data in the standby state.

[0146] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the non-transitory computer-readable storage media in the memory are used to store at least one instruction, which is executed by a processor to implement the methods in the embodiments of this application.

[0147] In some embodiments, the electronic device further includes a peripheral device interface and at least one peripheral device. The processor, memory, and peripheral device interface are connected via a bus or signal line. Each peripheral device is connected to the peripheral device interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes: a display screen, a camera, and audio circuitry. The peripheral device interface can be used to connect at least one I / O (Input / Output) related peripheral device to the processor and memory.

[0148] In some embodiments of this application, the processor, memory, and peripheral device interfaces are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor, memory, and peripheral device interfaces can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.

[0149] The electronic device structural block diagrams shown in the embodiments of this application do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0150] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An anomaly detection method applied to an inspection robot, characterized in that, include: The inspection robot's image acquisition device collects video streams along the inspection path; Based on the video stream, keyframes are sampled to extract multiple keyframes, and each keyframe is processed to obtain an input tensor. The image encoder of the pre-trained visual language large model extracts the image features of the input tensor to obtain a feature map, and the visual feature vector is obtained by processing the feature map. Calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and determine whether there is an abnormal target based on the cosine similarity. If the existence of abnormal targets is determined, an abnormal heat map is obtained based on the cosine similarity and the feature map to describe the distribution of abnormal targets; The location coordinates of each anomalous target are determined based on the aforementioned anomaly heatmap; The steps for constructing the normal feature prototype vector include: Multiple corpora describing the scene where the inspection robot is located as being in a normal state are obtained, and prompt words are constructed based on each corpus; wherein, the normal state means that there are no abnormal targets in the scene where the inspection robot is located; Each prompt word is input into the visual language big model, the prompt word is converted into a word sequence, and the text features of the word sequence are extracted by the text encoder of the visual language big model to obtain the corresponding text feature vectors. The normal feature prototype vector is obtained by fusing features based on all text feature vectors.

2. The anomaly detection method for an inspection robot according to claim 1, characterized in that, The process of obtaining the input tensor based on each keyframe includes: Scale the size of the keyframe to a preset input size; Center-crop the scaled keyframes to remove edge distortion; The keyframes after center clipping are converted into tensors and then normalized. Based on the pre-trained visual language large model, the mean and standard deviation of the pixel values ​​are used to standardize the normalized tensor, generating an input tensor of a preset dimension.

3. The anomaly detection method for an inspection robot according to claim 1, characterized in that, The calculation of the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and the determination of whether an abnormal target exists based on the cosine similarity, includes: The dot product operation is performed on the visual feature vector and the normal feature prototype vector to obtain the cosine similarity between the two vectors. The cosine similarity represents the degree of matching between the keyframe corresponding to the visual feature vector and the normal state. Compare the cosine similarity of each keyframe with the cosine similarity threshold. If the cosine similarity of the current keyframe and the consecutive preset number of keyframes before the current keyframe are all less than the cosine similarity threshold, then it is determined that there is an abnormal target in the current keyframe.

4. The anomaly detection method for an inspection robot according to claim 3, characterized in that, Before deploying the inspection robot, the method also includes: The inspection robot is deployed in a scene where there are no abnormal targets and the scene is in a normal state. The inspection robot collects normal environmental video streams of the scene in a normal state. Calculate the cosine similarity between the visual feature vector of each image frame in the normal environment video stream and the normal feature prototype vector, and calculate the mean and standard deviation of the cosine similarity. The cosine similarity threshold is calculated based on the mean and standard deviation of the cosine similarity.

5. The anomaly detection method for an inspection robot according to claim 1, characterized in that, The process of obtaining an anomaly heatmap based on the cosine similarity and the feature map to describe the distribution of abnormal targets includes: The objective function is constructed based on the difference between the cosine similarity of each keyframe and 1. Obtain the pixel value of each pixel position in each feature channel in the feature map, and calculate the gradient of the objective function with respect to each pixel value based on the backpropagation algorithm; Global average pooling is performed on each gradient to calculate the weights corresponding to each feature channel of the feature map. Based on the weights, each feature channel of the feature map is linearly weighted and superimposed. The weighted and superimposed feature map is then processed by an activation function to filter out negative values, resulting in a heat map value at each pixel location. The abnormal heat map is then constructed based on the heat map value at each pixel location.

6. An anomaly detection method for an inspection robot according to any one of claims 1-5, characterized in that, Determining the location coordinates of each anomalous target based on the anomalous heatmap includes: The abnormal heat map is binarized, and heat values ​​greater than the heat value threshold are set to 1, while heat values ​​less than or equal to the heat value threshold are set to 0. Determine the connected regions composed of pixels with a thermal value of 1 in the abnormal heatmap, determine the centroid of each connected region, and calculate the centroid coordinates. The depth value of the image acquisition device at the centroid coordinates is obtained, and the spatial coordinates of the centroid relative to the inspection robot are calculated based on the camera intrinsic parameter matrix of the image acquisition device and the depth value, so as to obtain the position coordinates of the abnormal target at the centroid position.

7. An anomaly detection device applied to an inspection robot, characterized in that, include: The image acquisition unit is used to acquire video streams along the inspection path through the image acquisition device of the inspection robot. The image processing unit is used to sample keyframes based on the video stream, extract multiple keyframes, and process each keyframe to obtain an input tensor. The feature extraction unit is used to extract image features from the input tensor through the image encoder of the pre-trained visual language large model to obtain a feature map, and to process the feature map to obtain a visual feature vector. An anomaly detection unit is used to calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and to determine whether there is an abnormal target based on the cosine similarity. If the presence of an abnormal target is determined, the anomaly detection unit is further configured to obtain an anomaly heatmap describing the distribution of the abnormal target based on the cosine similarity and the feature map processing. The anomaly detection unit is also used to determine the location coordinates of each anomaly target based on the anomaly heatmap. The steps for constructing the normal feature prototype vector include: Multiple corpora describing the scene where the inspection robot is located as being in a normal state are obtained, and prompt words are constructed based on each corpus; wherein, the normal state means that there are no abnormal targets in the scene where the inspection robot is located; Each prompt word is input into the visual language big model, the prompt word is converted into a word sequence, and the text features of the word sequence are extracted by the text encoder of the visual language big model to obtain the corresponding text feature vectors. The normal feature prototype vector is obtained by fusing features based on all text feature vectors.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images