Abnormality detection method and device applied to inspection robot, equipment and medium

Abstract

The invention discloses an anomaly detection method, device and equipment applied to an inspection robot and a medium, and relates to the technical field of robot autonomous navigation and computer vision crossing, and the method comprises the steps: collecting a video stream on an inspection path, carrying out the extraction to obtain a key frame, and carrying out the processing to obtain an input tensor; obtaining a feature map based on the input tensor through an image encoder of a pre-trained visual language large model, and processing the feature map to obtain a visual feature vector; calculating the cosine similarity between the visual feature vector and the normal feature prototype vector, and determining whether an abnormal target exists or not; and an abnormal thermodynamic diagram is obtained based on cosine similarity and feature map processing, and position coordinates of each abnormal target are determined. According to the method, the problem of converting anomaly detection into similarity measurement of the visual feature vector and the normal feature prototype vector is solved, so that dependence on negative sample training is avoided, the model does not need to learn all possible anomaly forms in advance, and the problem of missing detection caused by scarcity of negative samples is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot autonomous navigation and computer vision, and in particular to an abnormality detection method and device applied to a patrol robot, equipment and a medium. BACKGROUND

[0002] The patrol robot applied in industrial production, public safety and home service and the like needs to discover abnormal situations (such as accumulation of sundries in a corridor, damage to the appearance of equipment, leakage of liquid on the ground and the like) in a scene in time through the ability of “perception and understanding” of the environment state, which is crucial to guarantee production safety and maintain the order of the environment. The related technology generally collects image data in real time through the patrol robot, and realizes abnormality detection through the abnormality detection technology based on machine vision.

[0003] However, the related technology still has the following application problems: firstly, the existing mainstream abnormality detection algorithm (such as a classification model based on supervised learning) usually needs a large number of “positive samples” (normal images) and “negative samples” (abnormal images) for training. However, the data of the normal environment is easy to obtain, but the abnormal situations are often sporadic, sparse and various in form (i.e. long-tail distribution), and it is difficult to exhaust all possible abnormal samples for model training. Once an abnormal type not contained in the training set is encountered, the recognition rate of the traditional model will decrease significantly, resulting in missed detection. On the other hand, although the traditional unsupervised abnormality detection method does not need negative samples, it needs to “remember” the pixel distribution of the normal samples, and this method can only identify the statistical deviation at the pixel level and lacks the understanding of the scene semantics. When the environment changes, for example, the light changes, the angle is adjusted or dynamic objects unrelated to the task (such as a person walking normally in the corridor) appear, this kind of method is prone to false positives and needs to collect data again to retrain the model, and cannot understand the definition of “abnormality” in different scenes. On the other hand, the existing patrol robot cannot intuitively present the source of the abnormality to the user, and the patrol robot cannot understand natural language instructions of human beings, for example, the user cannot simply tell the robot to “check whether the ground is clean”, and this semantic gap limits the flexibility and ease of use of the robot in complex human-machine cooperation scenes.

[0004] To sum up, how to get rid of the dependence on negative sample training, utilize general semantic knowledge to realize abnormality detection in unknown and open scenes, and have good environmental adaptability and explainability is a key problem to be solved for the current mobile robot patrol technology. SUMMARY

[0005] The present application provides an abnormality detection method, device, equipment and medium applied to a patrol robot to solve the defects of the related technology, and the technical solution is as follows: In a first aspect, the application provides an anomaly detection method applied to a patrol robot, comprising: collecting a video stream on a patrol path through an image acquisition device of the patrol robot; sampling key frames based on the video stream, extracting a plurality of key frames, and processing an input tensor based on each key frame; extracting image features of the input tensor through an image encoder of a pre-trained visual language large model to obtain a feature map, and processing a visual feature vector based on the feature map; calculating a cosine similarity between the visual feature vector and a pre-constructed normal feature prototype vector, and determining whether there is an abnormal target according to the cosine similarity; in a case where it is determined that there is an abnormal target, processing an abnormal heat map for describing a distribution of the abnormal target based on the cosine similarity and the feature map; determining a position coordinate of each abnormal target based on the abnormal heat map.

[0006] In an optional solution of the first aspect, the processing of the input tensor based on each key frame comprises: scaling a size of the key frame to a preset input size; center cropping the scaled key frame to remove edge distortion; converting the center cropped key frame into a tensor and performing normalization processing; performing standardization processing on the normalized tensor based on a mean and a standard deviation of pixel values preset by the pre-trained visual language large model to generate an input tensor of a preset dimension size.

[0007] In an optional solution of the first aspect, the construction of the normal feature prototype vector comprises: obtaining a plurality of corpora for describing a normal state of a scene where the patrol robot is located, and constructing a prompt word based on each corpus; wherein the normal state indicates that there is no abnormal target in the scene where the patrol robot is located; inputting each prompt word into the visual language large model, converting the prompt word into a token sequence, extracting a text feature of the token sequence through a text encoder of the visual language large model, and respectively obtaining a corresponding text feature vector; performing feature fusion based on all text feature vectors to obtain the normal feature prototype vector.

[0008] In an optional solution of the first aspect, the calculation of the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and the determination of whether there is an abnormal target according to the cosine similarity, comprises: performing a dot product operation based on the visual feature vector and the normal feature prototype vector to obtain a cosine similarity between the two vectors, the cosine similarity representing a matching degree of the key frame corresponding to the visual feature vector with the normal state; comparing the cosine similarity of each key frame with a cosine similarity threshold, and if the cosine similarity of the current key frame and a continuous preset number of key frames before the current key frame are all less than the cosine similarity threshold, it is determined that the current key frame has an abnormal target.

[0009] In an optional implementation of the first aspect, before deploying the inspection robot, the method further includes: deploying the inspection robot in a scene where there is no abnormal target in a normal state, and collecting a normal environment video stream of the scene in the normal state by the inspection robot; calculating a cosine similarity of a visual feature vector of each image frame in the normal environment video stream with the normal feature prototype vector, and calculating a mean value and a standard deviation of the cosine similarity; calculating the cosine similarity threshold based on the mean value and the standard deviation of the cosine similarity.

[0010] In an optional implementation of the first aspect, the abnormal heat map used to describe the distribution of the abnormal target is obtained based on the cosine similarity and the feature map, including: constructing a target function based on a difference between the cosine similarity of each key frame and 1; obtaining pixel values of each pixel position in each feature channel in the feature map, and calculating gradients of the target function with respect to each pixel value based on a back propagation algorithm; performing global average pooling processing on each gradient to calculate a weight corresponding to each feature channel of the feature map; linearly weighting and superimposing each feature channel of the feature map based on the weight, processing the feature map after the weighting and superimposition by an activation function to filter out negative values, obtaining a heat value at each pixel position, and constructing the abnormal heat map based on the heat value at each pixel position.

[0011] In an optional implementation of the first aspect, the position coordinates of each abnormal target are determined based on the abnormal heat map, including: performing binaryzation processing on the abnormal heat map, setting a heat value greater than a heat value threshold to 1, and setting a heat value less than or equal to the heat value threshold to 0; determining a connected domain composed of pixels with a heat value of 1 in the abnormal heat map, determining a centroid of each connected domain, and calculating centroid coordinates; obtain a depth value of the image acquisition device at the centroid coordinate, calculate a spatial coordinate of the centroid relative to the inspection robot based on an intrinsic matrix of a camera of the image acquisition device and the depth value, and obtain a position coordinate of the abnormal target at the centroid position.

[0012] In a second aspect, the present application also provides an abnormality detection device applied to an inspection robot, comprising: an image acquisition unit configured to acquire a video stream on an inspection path through an image acquisition device of the inspection robot; an image processing unit configured to perform key frame sampling based on the video stream, extract a plurality of key frames, and obtain an input tensor based on processing of each key frame; a feature extraction unit configured to extract an image feature of the input tensor through an image encoder of a pre-trained visual language large model to obtain a feature map, and obtain a visual feature vector based on processing of the feature map; an abnormality detection unit configured to calculate a cosine similarity between the visual feature vector and a normal feature prototype vector constructed in advance, and determine whether an abnormal target exists according to the cosine similarity; In a case where it is determined that the abnormal target exists, the abnormality detection unit is further configured to obtain an abnormal heat map for describing a distribution condition of the abnormal target based on the cosine similarity and the feature map; The abnormality detection unit is further configured to determine a position coordinate of each abnormal target based on the abnormal heat map.

[0013] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the first aspect or any one of the implementation manners of the first aspect.

[0014] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method provided in the first aspect or any one of the implementation manners of the first aspect.

[0015] The technical scheme provided by the present application has at least the following beneficial effects: The abnormality detection method applied to the inspection robot provided by the present application extracts a visual feature vector through a direct call of an image encoder of a pre-trained visual language large model, and converts abnormality detection into a similarity measurement problem between the visual feature vector and a normal feature prototype vector, which can avoid the dependence on negative sample (abnormal image) training, so that the model does not need to learn all possible abnormal morphologies in advance, and can effectively identify unknown abnormal types outside the training set, thereby solving the problem of missed detection caused by the scarcity of negative samples.

[0016] Secondly, since the extracted visual feature vector contains rich semantic information, the detection process is based on high-level scene understanding rather than pixel-level statistics, thereby significantly enhancing the robustness to dynamic changes in the environment, effectively distinguishing real anomalies from interference such as light changes, perspective differences, or irrelevant dynamic objects, greatly reducing the false positive rate of traditional unsupervised methods due to the lack of semantic understanding, and improving the adaptability and reliability in complex environments.

[0017] Finally, by generating an abnormal heat map that visually displays the area where the abnormal target is located and outputting accurate position coordinates, a clear visual explanation of the detection result is provided, enabling users to quickly locate and understand the source of the anomaly, thereby providing a feasible technical support for intelligent inspection. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0019] Figure 1 is a flowchart of an anomaly detection method for a patrol robot provided by an embodiment of the present application; Figure 2 is a structural schematic diagram of an anomaly detection device for a patrol robot provided by an embodiment of the present application; Figure 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely in the following with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] The terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-described drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but can optionally include steps or modules not listed, or can optionally include other steps or modules inherent to the process, method, product or device.

[0022] It should be noted that the terms "first" and "second" in the present application are merely to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first" and "second" can be interchanged in a specific order or sequence as allowed. It should be understood that the objects distinguished by "first" and "second" can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0023] The present application will be described in detail below with reference to specific embodiments.

[0024] Next, an abnormality detection method applied to a patrol robot provided by an embodiment of the present application will be introduced. For details, please refer to Figure 1 Figure 1 , Figure 1 Fig. 1 shows a flowchart of an abnormality detection method applied to a patrol robot provided by an embodiment of the present application. As shown in Figure 1 , the method comprises the following steps: S101, collecting a video stream on a patrol path by an image acquisition device of a patrol robot; S102, performing key frame sampling based on the video stream to extract a plurality of key frames, and processing an input tensor based on each key frame; S103, extracting image features of the input tensor by an image encoder of a pre-trained visual language large model to obtain a feature map, and processing a visual feature vector based on the feature map; S104, calculating a cosine similarity between the visual feature vector and a pre-constructed normal feature prototype vector, and determining whether there is an abnormal target according to the cosine similarity; S105, in the case where it is determined that there is an abnormal target, processing an abnormal heat map for describing a distribution situation of the abnormal target based on the cosine similarity and the feature map; S106, determining a position coordinate of each abnormal target based on the abnormal heat map.

[0025] In some embodiments, the patrol robot in S101 can be a Unitree B1 industrial four-legged robot platform, and the image acquisition device carried by the patrol robot can include but is not limited to an RGB camera, a depth camera and various lenses, for example, an Intel RealSense D435i depth camera can be selected, and the present application is not limited thereto.

[0026] Specifically, the video stream resolution and frame rate of the image acquisition device can be set in advance, for example, the video stream resolution is set to 640x480 and the frame rate is set to 30fps.

[0027] ​Specifically, the inspection robot can control the image acquisition device carried by the inspection robot to collect an environmental video stream on the current inspection path in real time in response to an inspection instruction issued by a user.

[0028] In some embodiments, in S102, in order to balance between the calculation accuracy and the real-time performance of the system, the system does not process all image frames in the video stream frame by frame, but adopts an adaptive key frame extraction strategy based on the motion state, that is, key frames are extracted from the video stream for processing according to the motion state of the robot.

[0029] Specifically, the key frames can be collected from the video stream based on a preset sampling rate, the sampling rate can be reduced when the inspection robot moves fast to reduce the interference of blurred frames, and the sampling rate can be increased when the inspection robot moves slowly or is stationary to obtain clear details.

[0030] For example, the speedometer data of the robot chassis can be read, if the speed is greater than 0.5 m / s, an image is read as a key frame every 5 frames; if the speed is less than 0.5 m / s, each frame is read frame by frame, and each frame is regarded as a key frame to reduce the influence of motion blur on feature extraction.

[0031] In some embodiments, the process of obtaining an input tensor based on each key frame in S102 includes: S1021, scaling the size of the key frame to a preset input size; The preset input size is specifically a standard input size of an image encoder of a visual language large model (Contrastive Language-Image Pre-training, CLIP), for example, the resize function of OpenCV can be called to scale the captured image to 224 pixels.

[0032] S1022, center cropping the scaled key frame to remove edge distortion.

[0033] For example, the image center 224 224 area of the key frame can be intercepted.

[0034] S1023, converting the center cropped key frame into a PyTorch tensor and performing normalization processing, dividing the pixel value by 255 to normalize it to the interval [0, 1].

[0035] S1024, based on the mean and standard deviation of the pixel value preset by the pre-trained visual language large model, the normalized tensor is standardized to generate an input tensor of a preset dimension size.

[0036] For example, by standardizing using preset mean (0.4814, 0.4578, 0.4082) and standard deviation (0.2686, 0.2613, 0.2757), an input tensor with preset dimensions [1, 3, 224, 224] is obtained.

[0037] Specifically, the visual language big model provided in this application includes a Vision Transformer image encoder and a Text Transformer text encoder.

[0038] For example, the specific model selected for the large visual language model could be ViT-B / 32.

[0039] Furthermore, the image encoder based on the large visual language model performs step S103, feeding the input tensor into the image encoder and extracting image features through the image encoder to obtain a feature map; Further, L2 normalization is performed on the feature map to obtain the visual feature vector. The dimension of the visual feature vector can be 512.

[0040] In some embodiments, normal feature prototype vectors can be constructed by inputting a large corpus into a large visual language model. The specific construction steps include: S201, Obtain multiple corpora describing the scene where the inspection robot is located as being in a normal state, and construct prompt words based on each corpus; Specifically, each corpus consists of natural language text describing the scene in which the inspection robot is located as being in a normal state, wherein the normal state means that there are no abnormal targets in the scene in which the inspection robot is located.

[0041] The purpose of using multiple corpora instead of a single corpus is to cover different aspects of normal scenarios and improve the system's generalization ability.

[0042] For example, the prompt words may include: "A photo of a clean server room floor" "A tidy corridor with no obstacles" "empty passage" "Standard industrial floor".

[0043] S202, each prompt word is input into the visual language big model, the prompt word is converted into a word sequence (i.e., a token sequence), and the text features of the word sequence are extracted by the text encoder of the visual language big model to obtain the corresponding text feature vectors.

[0044] S203, perform feature fusion based on all text feature vectors to obtain the normal feature prototype vector.

[0045] Specifically, feature fusion based on all text feature vectors can be understood as calculating the arithmetic mean of all text feature vectors. The result of feature fusion can then be further normalized to obtain the normal feature prototype vector. .

[0046] In some embodiments, S104 specifically includes: S1041, perform a dot product operation based on the visual feature vector and the normal feature prototype vector to obtain the cosine similarity between the two vectors.

[0047] Specifically, cosine similarity can be denoted as S. Cosine similarity is used to intuitively represent the degree of matching between the keyframe corresponding to the visual feature vector and the scene in the normal state. It can be understood as the semantic similarity between the visual feature vector and the normal feature prototype vector. The higher the cosine similarity, the lower the probability of an abnormal target in the keyframe, and the more the keyframe conforms to the description of "neat and clean". Otherwise, the higher the probability of an abnormal target in the keyframe.

[0048] S1042, compare the cosine similarity of each key frame with the cosine similarity threshold. If the cosine similarity of the current key frame and the consecutive preset number of key frames before the current key frame are all less than the cosine similarity threshold, then it is determined that there is an abnormal target in the current key frame.

[0049] Specifically, if This indicates that the current keyframe corresponds to a scene in a normal state; like This retrieves the keyframe sequence of the current keyframe and the previous preset number of keyframes. For example, if the preset number is set to 5, the keyframe sequence will be retrieved in chronological order. ,in For the current keyframe, For keyframes preceding the current keyframe, if the cosine similarity of 5 consecutive frames is less than the cosine similarity threshold, then the current keyframe is considered to be... An abnormal target exists.

[0050] In some embodiments, before deploying the inspection robot, a cosine similarity threshold can be adaptively calculated from video data collected by the inspection robot under normal conditions, including: S301, The inspection robot is deployed in a scene where there are no abnormal targets and the scene is in a normal state, and the inspection robot collects a normal environmental video stream of the scene in a normal state. S302, calculate the cosine similarity between the visual feature vector of each image frame in the normal environment video stream and the normal feature prototype vector, and calculate the mean of the cosine similarity. and standard deviation ; S303, the cosine similarity threshold is calculated based on the mean and standard deviation of the cosine similarity. Apply the formula: ; For example, mean The standard deviation is 0.85. The cosine similarity threshold is calculated to be 0.03. It is 0.76.

[0051] In some embodiments, if an anomalous target is determined to exist, step S105 is performed to precisely locate the image region causing the reduced cosine similarity, specifically including: S1051, based on the cosine similarity of each keyframe The objective function is constructed from the difference between 1 and 1. , is represented as: ; Wherein, objective function Cosine similarity The complement of.

[0052] S1052, obtain the pixel value of each pixel position in each feature channel in the feature map, and calculate the gradient of the objective function with respect to each pixel value based on the backpropagation algorithm.

[0053] Specifically, the feature map can be represented as A, and the feature map A's... Each feature channel is represented as At each pixel location in feature map A The pixel value on each feature channel k is represented as The calculated gradient is .

[0054] S1053, Perform global average pooling on each gradient to calculate the weights corresponding to each feature channel of the feature map, applying the formula:

[0055] in, The weight is represented by , and N represents the total number of pixels in the feature map.

[0056] S1054, based on the weights For each feature channel of the feature map Linear weighted summation is performed, and the resulting feature map is processed using the ReLU activation function to filter out negative values ​​(i.e., retain regions that positively contribute to anomaly detection), obtaining the heatmap value at each pixel location. The anomaly heatmap is then constructed based on the heatmap values ​​at each pixel location. , is represented as:

[0057] Furthermore, it is also possible to analyze abnormal heatmaps. Post-processing is performed, with the input size as... Keyframes, spatial dimensions are Taking a feature map with 512 channels as an example, the generated... The size of the anomaly heatmap was upsampled using bilinear interpolation. The original input size is used, and OpenCV's Jet color map is applied to overlay each heat value onto the original grayscale image to form an intuitive anomaly indicator.

[0058] Understandably, the higher the heat value, the higher the probability that there is an abnormal target (obstacle) at the corresponding location. A mapping relationship between heat values ​​and RGB colors can be constructed, and the distribution of abnormal targets can be intuitively reflected on the abnormal heat map through color.

[0059] Furthermore, the steps in S106 include: S1061, the abnormal heat map is binarized, and heat values ​​greater than the heat value threshold are set to 1, and heat values ​​less than or equal to the heat value threshold are set to 0. S1062, determine the connected components formed by pixels with a thermal value of 1 in the abnormal heatmap, determine the centroid of each connected component and calculate the centroid coordinates. ; A connected component is an image region consisting of pixels with the same pixel value and adjacent positions.

[0060] S1063, obtain the depth value Z of the image acquisition device at the centroid coordinates, calculate the spatial coordinates of the centroid relative to the inspection robot based on the camera intrinsic parameter matrix of the image acquisition device and the depth value, and obtain the position coordinates of the abnormal target at the centroid position.

[0061] Specifically, the camera intrinsic parameter matrix includes focal length. He Guangxin The spatial coordinates of the centroid relative to the inspection robot are calculated based on the camera intrinsic parameter matrix, using the following formula: ; ; This allows us to obtain the spatial coordinates of the centroid relative to the inspection robot. That is, the location coordinates of the abnormal target.

[0062] In some embodiments, after obtaining the location coordinates of the abnormal target, a ROS (Robot Operating System) message can be constructed by the program on the inspection robot. This message contains the abnormal target type label "UnknownAnomaly", the image data and spatial coordinates of the abnormal target, and is sent to the user's terminal for the user to view.

[0063] In some embodiments, the navigation planning module of the inspection robot can mark the corresponding location as an obstacle in the local cost map of the robot navigation based on the spatial coordinates of the abnormal target, and avoid the area corresponding to the obstacle during the subsequent inspection path planning process. If the area corresponding to the obstacle completely blocks the inspection path, the inspection robot stops moving and sends an alarm information containing the bounding box of the abnormal target and its location coordinates to the user terminal via the network for manual confirmation and processing.

[0064] The following are apparatus embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of this application.

[0065] Please see below. Figure 2 The following is a schematic diagram of an anomaly detection device for an inspection robot, provided as an exemplary embodiment of this application. The device includes: The image acquisition unit is used to acquire video streams along the inspection path through the image acquisition device of the inspection robot. The image processing unit is used to perform keyframe sampling based on the video stream, extract multiple keyframes, and process each keyframe to obtain an input tensor. The feature extraction unit is used to extract image features from the input tensor through the image encoder of the pre-trained visual language large model to obtain a feature map, and to process the feature map to obtain a visual feature vector. An anomaly detection unit is used to calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and to determine whether there is an abnormal target based on the cosine similarity. If the presence of an abnormal target is determined, the anomaly detection unit is also used to obtain an anomaly heatmap describing the distribution of the abnormal target based on the cosine similarity and the feature map processing. The anomaly detection unit is also used to determine the location coordinates of each anomaly target based on the anomaly heatmap.

[0066] It should be noted that the apparatus provided in the above embodiments, when executing an anomaly detection method applied to an inspection robot, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process is detailed in the method embodiments, which will not be repeated here.

[0067] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0068] Please see Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0069] like Figure 3 As shown, the electronic device includes a processor and a memory.

[0070] In this embodiment, the processor is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array).

[0071] A processor can also include a main processor and a coprocessor. The main processor is used to process data in the wake-up state and is also called the CPU (Central Processing Unit). The coprocessor is a low-power processor used to process data in the standby state.

[0072] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the non-transitory computer-readable storage media in the memory are used to store at least one instruction, which is executed by a processor to implement the methods in the embodiments of this application.

[0073] In some embodiments, the electronic device further includes a peripheral device interface and at least one peripheral device. The processor, memory, and peripheral device interface are connected via a bus or signal line. Each peripheral device is connected to the peripheral device interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes: a display screen, a camera, and audio circuitry. The peripheral device interface can be used to connect at least one I / O (Input / Output) related peripheral device to the processor and memory.

[0074] In some embodiments of this application, the processor, memory, and peripheral device interfaces are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor, memory, and peripheral device interfaces can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.

[0075] The electronic device structural block diagrams shown in the embodiments of this application do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0076] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An anomaly detection method applied to an inspection robot, characterized in that, include: The inspection robot's image acquisition device collects video streams along the inspection path; Based on the video stream, keyframes are sampled to extract multiple keyframes, and each keyframe is processed to obtain an input tensor. The image encoder of the pre-trained visual language large model extracts the image features of the input tensor to obtain a feature map, and the visual feature vector is obtained by processing the feature map. Calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and determine whether there is an abnormal target based on the cosine similarity. If the existence of abnormal targets is determined, an abnormal heat map is obtained based on the cosine similarity and the feature map to describe the distribution of abnormal targets; The location coordinates of each abnormal target are determined based on the aforementioned anomaly heatmap.

2. The anomaly detection method for an inspection robot according to claim 1, characterized in that, The process of obtaining the input tensor based on each keyframe includes: Scale the size of the keyframe to a preset input size; Center-crop the scaled keyframes to remove edge distortion; The keyframes after center clipping are converted into tensors and then normalized. Based on the pre-trained visual language large model, the mean and standard deviation of the pixel values ​​are used to standardize the normalized tensor, generating an input tensor of a preset dimension.

3. The anomaly detection method for an inspection robot according to claim 1, characterized in that, The steps for constructing the normal feature prototype vector include: Multiple corpora describing the scene where the inspection robot is located as being in a normal state are obtained, and prompt words are constructed based on each corpus; wherein, the normal state means that there are no abnormal targets in the scene where the inspection robot is located; Each prompt word is input into the visual language big model, the prompt word is converted into a word sequence, and the text features of the word sequence are extracted by the text encoder of the visual language big model to obtain the corresponding text feature vectors. The normal feature prototype vector is obtained by fusing features based on all text feature vectors.

4. The anomaly detection method for an inspection robot according to claim 3, characterized in that, The calculation of the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and the determination of whether an abnormal target exists based on the cosine similarity, includes: The dot product operation is performed on the visual feature vector and the normal feature prototype vector to obtain the cosine similarity between the two vectors. The cosine similarity represents the degree of matching between the keyframe corresponding to the visual feature vector and the normal state. Compare the cosine similarity of each keyframe with the cosine similarity threshold. If the cosine similarity of the current keyframe and the consecutive preset number of keyframes before the current keyframe are all less than the cosine similarity threshold, then it is determined that there is an abnormal target in the current keyframe.

5. The anomaly detection method for an inspection robot according to claim 4, characterized in that, Before deploying the inspection robot, the method also includes: The inspection robot is deployed in a scene where there are no abnormal targets and the scene is in a normal state. The inspection robot collects normal environmental video streams of the scene in a normal state. Calculate the cosine similarity between the visual feature vector of each image frame in the normal environment video stream and the normal feature prototype vector, and calculate the mean and standard deviation of the cosine similarity. The cosine similarity threshold is calculated based on the mean and standard deviation of the cosine similarity.

6. The anomaly detection method for an inspection robot according to claim 1, characterized in that, The process of obtaining an anomaly heatmap based on the cosine similarity and the feature map to describe the distribution of abnormal targets includes: The objective function is constructed based on the difference between the cosine similarity of each keyframe and 1. Obtain the pixel value of each pixel position in each feature channel in the feature map, and calculate the gradient of the objective function with respect to each pixel value based on the backpropagation algorithm; Global average pooling is performed on each gradient to calculate the weights corresponding to each feature channel of the feature map. Based on the weights, each feature channel of the feature map is linearly weighted and superimposed. The weighted and superimposed feature map is then processed by an activation function to filter out negative values, resulting in a heat map value at each pixel location. The abnormal heat map is then constructed based on the heat map value at each pixel location.

7. An anomaly detection method for an inspection robot according to any one of claims 1-6, characterized in that, Determining the location coordinates of each anomalous target based on the anomalous heatmap includes: The abnormal heat map is binarized, and heat values ​​greater than the heat value threshold are set to 1, while heat values ​​less than or equal to the heat value threshold are set to 0. Determine the connected regions composed of pixels with a thermal value of 1 in the abnormal heat map, determine the centroid of each connected region, and calculate the centroid coordinates. The depth value of the image acquisition device at the centroid coordinates is obtained, and the spatial coordinates of the centroid relative to the inspection robot are calculated based on the camera intrinsic parameter matrix of the image acquisition device and the depth value, so as to obtain the position coordinates of the abnormal target at the centroid position.

8. An anomaly detection device for use in inspection robots, characterized in that, include: The image acquisition unit is used to acquire video streams along the inspection path through the image acquisition device of the inspection robot. The image processing unit is used to perform keyframe sampling based on the video stream, extract multiple keyframes, and process each keyframe to obtain an input tensor. The feature extraction unit is used to extract image features from the input tensor through the image encoder of the pre-trained visual language large model to obtain a feature map, and to process the feature map to obtain a visual feature vector. An anomaly detection unit is used to calculate the cosine similarity between the visual feature vector and the pre-constructed normal feature prototype vector, and to determine whether there is an abnormal target based on the cosine similarity. If the presence of an abnormal target is determined, the anomaly detection unit is also used to obtain an anomaly heatmap describing the distribution of the abnormal target based on the cosine similarity and the feature map processing. The anomaly detection unit is also used to determine the location coordinates of each anomaly target based on the anomaly heatmap.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images