Door and window opening and closing detection method and device based on robot inspection, medium and equipment

Abstract

The application discloses a door and window opening and closing detection method and device based on robot inspection, a medium and equipment, comprising collecting reference images and to-be-detected images of detection points based on a patrol robot; aligning the to-be-detected images with the reference images, performing background difference, and obtaining a foreground target binary image; performing image processing on the foreground target binary image, and obtaining a suspicious area frame; mapping the suspicious area frame to the reference images and the aligned to-be-detected images, respectively obtaining reference areas and to-be-detected areas; performing feature extraction on the reference areas and the to-be-detected areas, respectively obtaining reference area image feature vectors and to-be-detected area image feature vectors; calculating cosine similarity of the two image feature vectors; judging whether a door and window is opened or not according to the cosine similarity, and generating door and window opening and closing information. The application effectively overcomes the problems of high modification cost and low detection robustness in door and window opening and closing detection, and greatly improves the application generalization of door and window opening and closing detection.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium and equipment for detecting the opening and closing of doors and windows based on robot inspection. Background Technology

[0002] To prevent theft and other losses in warehouses or other public places, it is necessary to inspect and check the doors and windows to ensure they are all securely closed. Generally, leaving doors and windows open increases the risk of theft.

[0003] In existing technologies, the detection of door and window opening / closing involves several methods. One method involves installing additional mechanical or electronic devices at the door or window, where opening and closing triggers different states of these devices, and the openness of the door or window is determined based on these states. Another method involves installing cameras in front of the door or window to capture real-time images, performing image processing such as optical flow analysis or door frame edge detection, and determining whether the door or window is open based on the image processing results. However, adding mechanical or electronic devices or installing cameras is costly and has limitations in deployment and promotion, resulting in poor generalization. Another method involves using robots equipped with 2D LiDAR to collect radar information directly facing the door or window, and determining whether the door or window is open based on the collected information. Yet another method involves affixing signs to the door or window, using robots to collect image information, and using deep learning to detect the position of the signs in the collected images, determining whether the door or window is open based on the sign's position. However, these detection methods suffer from low robustness and poor application generalization. Summary of the Invention

[0004] This invention provides a method, apparatus, medium, and equipment for detecting door and window switches based on robot inspection, in order to solve the problems of high modification costs, poor application generalization, and low detection robustness in the existing technology for detecting door and window switches.

[0005] A method for detecting the opening and closing of doors and windows based on robot inspection, the method comprising:

[0006] The patrol robot collects reference images and images to be detected at the detection points;

[0007] The SuperPoint and SuperGlue algorithms are used to align the image to be detected at the detection point with the reference image at the detection point to obtain the image to be detected after the detection point alignment.

[0008] The reference image and the image to be detected after the detection points are aligned are input into a Gaussian mixture model (GMM) for background subtraction to obtain a binary image of the foreground target.

[0009] Image processing is performed on the binary image of the foreground target to obtain a suspicious region bounding box;

[0010] mapping the suspicious region frame to the reference image to obtain a reference region, and mapping the suspicious region frame to the aligned to-be-detected image to obtain a to-be-detected region;

[0011] extracting features of the reference region by using a deep neural network to obtain a reference region image feature vector, and extracting features of the to-be-detected region by using a deep neural network to obtain a to-be-detected region image feature vector;

[0012] calculating a cosine similarity between the reference region image feature vector and the to-be-detected region image feature vector;

[0013] judging whether the door and window are open or not according to the cosine similarity to generate door and window opening and closing information.

[0014] Optionally, the to-be-detected image of the detection point and the reference image of the detection point are aligned by using a SuperPoint and a SuperGlue algorithm to obtain the aligned to-be-detected image of the detection point, and the method comprises:

[0015] inputting the reference image of the detection point into a first SuperPoint model to extract feature point information of the reference image, and inputting the to-be-detected image into a second SuperPoint model to extract feature point information of the to-be-detected image;

[0016] inputting the feature point information of the reference image and the feature point information of the to-be-detected image into a SuperGlue model to generate feature matching information of the reference image and the to-be-detected image;

[0017] aligning the to-be-detected image to the reference image according to the feature matching information to obtain the aligned to-be-detected image.

[0018] Optionally, the number of Gaussian kernels of the Gaussian Mixture Model (GMM) is 5, and a background updating operation of the Gaussian Mixture Model (GMM) is in a closed state in difference calculation.

[0019] Optionally, the image processing on the foreground target binary image to obtain the suspicious region frame comprises:

[0020] performing morphological filtering on the foreground target binary image to obtain a filtered foreground target binary image;

[0021] performing hole filling on the filtered foreground target binary image to obtain a hole-filled foreground target binary image;

[0022] performing contour extraction on the hole-filled foreground target binary image to obtain a suspicious target region of the foreground target binary image;

[0023] The suspicious region frame is obtained by performing a maximum bounding box operation on the suspicious region of the foreground object binary image.

[0024] Optionally, one foreground object binary image includes at least one suspicious region frame.

[0025] Optionally, the door and window opening and closing information is generated by judging whether the door and window are opened according to the cosine similarity.

[0026] The minimum similarity is initialized as 1.

[0027] Each suspicious region frame in the foreground object binary image is traversed, and the cosine similarity of the suspicious region frame is compared with the minimum similarity.

[0028] If the cosine similarity of the suspicious region frame is less than the minimum similarity, the minimum similarity is updated as the cosine similarity of the suspicious region frame.

[0029] Until all the suspicious region frames in the foreground object binary image are traversed, the minimum similarity is compared with a preset similarity threshold.

[0030] If the minimum similarity is less than the preset similarity threshold, the door and window in the to-be-detected image are in an open state, otherwise, the door and window in the to-be-detected image are in a closed state.

[0031] Optionally, the preset similarity threshold is 0.4.

[0032] A door and window opening and closing detection method based on robot patrol, the device comprises:

[0033] An image acquisition module is configured to acquire a reference image of a detection point and a to-be-detected image of the detection point based on a patrol robot.

[0034] An alignment module is configured to align the to-be-detected image of the detection point and the reference image of the detection point by using a SuperPoint and a SuperGlue algorithm, to obtain an aligned to-be-detected image of the detection point.

[0035] A difference module is configured to input the reference image and the aligned to-be-detected image of the detection point into a Gaussian Mixture Model (GMM) to perform background difference, to obtain a foreground object binary image.

[0036] An image processing module is configured to perform image processing on the foreground object binary image, to obtain a suspicious region frame.

[0037] A mapping module is configured to map the suspicious region frame to the reference image to obtain a reference region, and map the suspicious region frame to the aligned to-be-detected image of the detection point to obtain a to-be-detected region.

[0038] a feature extraction module configured to extract features of the reference region by using a deep neural network to obtain a reference region image feature vector, and extract features of the to-be-detected region by using a deep neural network to obtain a to-be-detected region image feature vector;

[0039] a similarity calculation module configured to calculate a cosine similarity between the reference region image feature vector and the to-be-detected region image feature vector;

[0040] a generation module configured to determine whether a door or window is open according to the cosine similarity, and generate door and window opening and closing information.

[0041] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the door and window opening and closing detection method based on robot inspection as described above.

[0042] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the door and window opening and closing detection method based on robot inspection as described above when executing the computer program.

[0043] The embodiment of the present application collects a reference image of a detection point and a to-be-detected image of the detection point based on a patrol robot; a SuperPoint and a SuperGlue algorithm are used to align the to-be-detected image of the detection point with the reference image of the detection point to obtain an aligned to-be-detected image of the detection point; the reference image and the aligned to-be-detected image of the detection point are input into a Gaussian Mixture Model (GMM) for background difference to obtain a foreground target binary image; the foreground target binary image is processed to obtain a suspicious region frame; the suspicious region frame is mapped to the reference image to obtain a reference region, and the suspicious region frame is mapped to the aligned to-be-detected image to obtain a to-be-detected region; a deep neural network is used to extract features of the reference region to obtain a reference region image feature vector, and a deep neural network is used to extract features of the to-be-detected region to obtain a to-be-detected region image feature vector; a cosine similarity between the reference region image feature vector and the to-be-detected region image feature vector is calculated; whether a door or window is open is determined according to the cosine similarity, and door and window opening and closing information is generated, thereby greatly improving the application generalization of door and window opening and closing detection and effectively overcoming the problems of high modification cost and low detection robustness in door and window opening and closing detection. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0045] Figure 1 is a flow chart of a door and window switch detection method based on robot patrol provided by an embodiment of the present application;

[0046] Figure 2 is a structural schematic diagram of a door and window switch detection device based on robot patrol provided by an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the scope of protection of the present application.

[0049] The door and window switch detection method based on robot patrol provided by the embodiments of the present application collects a reference image of a detection point and a to-be-detected image of the detection point based on a patrol robot; adopts a SuperPoint and a SuperGlue algorithm to align the to-be-detected image of the detection point with the reference image of the detection point, to obtain an aligned to-be-detected image of the detection point; inputs the reference image and the aligned to-be-detected image of the detection point into a Gaussian Mixture Model (GMM) for background difference, to obtain a foreground target binary image; performs image processing on the foreground target binary image, to obtain a suspicious region frame; maps the suspicious region frame to the reference image to obtain a reference region, and maps the suspicious region frame to the aligned to-be-detected image to obtain a to-be-detected region; adopts a deep neural network to extract features of the reference region, to obtain a reference region image feature vector, and adopts the deep neural network to extract features of the to-be-detected region, to obtain a to-be-detected region image feature vector; calculates a cosine similarity of the reference region image feature vector and the to-be-detected region image feature vector; judges whether a door and window is opened or not according to the cosine similarity, to generate door and window switch information, which greatly improves the application generalization of door and window switch detection, and effectively overcomes the problems of high transformation cost and low detection robustness in door and window switch detection.

[0050] The robot-based door and window opening and closing detection method provided by the embodiment will be described in detail as follows. Figure 1 As shown in the method, the robot-based door and window opening and closing detection method comprises the following steps.

[0051] In step S101, the patrol robot collects a reference image of a detection point and a to-be-detected image of the detection point.

[0052] Here, the reference image refers to an image of a pre-set door and window closing state, and the to-be-detected image refers to an image collected by the robot at a fixed shooting point during the patrol. The patrol robot is used to collect the reference image and the to-be-detected image at a designated detection point, and the detection point collection reference image and the to-be-detected image are further provided to the SuperPoint and SuperGlue algorithms for image alignment.

[0053] Optionally, as a preferred example of the present application, the patrol robot collects a reference image of a detection point and a to-be-detected image of the detection point, which comprises the following steps.

[0054] Setting the patrol route, the fixed shooting point and the shooting parameters: according to the use scene, the prior knowledge is used to set the patrol route that can cover the door and window to be detected, set the detection point in the patrol route, and set the corresponding camera parameters, including but not limited to the shooting angle, the exposure time and the angle, etc.

[0055] Collecting the reference image: according to the prior knowledge, the patrol robot is used to patrol the patrol route for several times, preferably 5 times, to obtain 5 door and window images of the detection point, and select 1 most suitable door and window closing image as the reference image of the detection point; wherein the most suitable door and window closing image refers to the door and window closing image without any abnormal interference.

[0056] Collecting the to-be-detected image: the patrol robot is controlled to run repeatedly in the patrol route to obtain the to-be-detected image of the detection point.

[0057] In step S102, the SuperPoint and SuperGlue algorithms are used to align the to-be-detected image of the detection point with the reference image of the detection point, to obtain the aligned to-be-detected image of the detection point.

[0058] Here, due to the possible error in the robot patrol navigation, the photos taken at each fixed shooting point may have some slight position or angle deviation. In order to reduce the influence of such deviation on the subsequent image background difference analysis, the embodiment adopts an image calibration alignment method to align the pictures. Optionally, as a preferred example of the present application, the step S102 of aligning the to-be-detected image of the detection point with the reference image of the detection point by using the SuperPoint and SuperGlue algorithm includes:

[0059] In step S201, the reference image of the detection point is input into the first SuperPoint model to extract the feature point information of the reference image, and the to-be-detected image is input into the second SuperPoint model to extract the feature point information of the to-be-detected image.

[0060] In step S202, the feature point information of the reference image and the feature point information of the to-be-detected image are input into the SuperGlue model to generate feature matching information of the reference image and the to-be-detected image.

[0061] In step S203, according to the feature matching information, the to-be-detected image is aligned to the reference image to obtain an aligned to-be-detected image.

[0062] Here, the embodiment adopts the image collected by the patrol robot as the training data set of the SuperPoint model SuperPoint, and adopts unsupervised learning as the training method of the SuperPoint model SuperPoint. The embodiment adopts a network open source data set as the training data set of the SuperGlue model SuperGlue. Optionally, the embodiment adopts the feature point position and the feature vector as the feature point information.

[0063] In step S103, the reference image and the aligned to-be-detected image of the detection point are input into the Gaussian Mixture Model GMM for background difference to obtain a foreground target binary image.

[0064] Optionally, as a preferred example of the present application, the number of Gaussian kernels of the Gaussian Mixture Model GMM is set to 5; and the background update operation of the Gaussian Mixture Model GMM is set to be in an off state in the difference calculation, that is, the background will not be automatically updated after each background difference.

[0065] In step S104, the foreground target binary image is subjected to image processing to obtain a suspicious area frame.

[0066] In this embodiment, the reference image and the aligned image to be detected are input into a Gaussian Mixture Model (GMM) for background difference to obtain a corresponding foreground target binary image. The foreground target in the binary image is a suspicious target for determining whether the door is open. Optionally, as a preferred example of the present application, the image processing of the foreground target binary image in step S104 includes:

[0067] In step S301, the foreground target binary image is subjected to morphological filtering to obtain a filtered foreground target binary image.

[0068] In step S302, the filtered foreground target binary image is subjected to hole filling to obtain a hole-filled foreground target binary image.

[0069] In step S303, the hole-filled foreground target binary image is subjected to contour extraction to obtain a suspicious target region of the foreground target binary image.

[0070] In step S304, a maximum bounding box operation is performed on the suspicious target region of the foreground target binary image to obtain a suspicious region frame.

[0071] It should be understood that the foreground target binary image obtained by the background difference through the Gaussian Mixture Model (GMM) includes at least one suspicious region frame.

[0072] In step S105, the suspicious region frame is mapped to the reference image to obtain a reference region, and the suspicious region frame is mapped to the aligned image to be detected to obtain a to-be-detected region.

[0073] In this embodiment, the suspicious region frame is mapped to obtain a reference region when mapped to the reference image, and a to-be-detected region when mapped to the aligned image to be detected. One suspicious region frame can map a pair of reference region and to-be-detected region, and the reference region and the to-be-detected region are corresponding. Then, the state of the door and window switch is found by feature extraction and comparison of the reference region and the to-be-detected region.

[0074] In step S106, a deep neural network is used to extract features of the reference region to obtain a reference region image feature vector, and a deep neural network is used to extract features of the to-be-detected region to obtain a to-be-detected region image feature vector.

[0075] Optionally, as a preferred example of the present application, a deep learning neural network is used for feature extraction in this embodiment. In training the deep neural network, unsupervised contrast learning is used, the loss function uses CircleLoss, and the training data set in the training process uses a network open source data set imagenet, which is preprocessed, including:

[0076] In step S401, each image in the network open source dataset imagenet is traversed to obtain an original image img_src.

[0077] In step S402, image transformation is performed on the original image img_src to obtain a positive image img_pos.

[0078] In step S403, other images are scaled and randomly pasted on the original image to obtain a negative image img_neg.

[0079] In step S404, the labels of the original image img_src and the positive image img_pos are set to the same label, and the label of the negative image img_neg is set to a different label from that of the original image img_src.

[0080] In step S405, until each image in the network open source dataset imagenet is traversed, a dataset for deep neural network unsupervised learning is obtained.

[0081] Here, if the label Label of the original image img_src is set to 1, the label of the positive image img_pos is set to 1, and the label of the negative image img_neg is set to a number other than 1.

[0082] Optionally, in step S402, image transformation is performed on the original image, including but not limited to jitter processing Jitter, scaling processing Scale, and rotation processing Rotate on the original image img_src.

[0083] In step S107, the cosine similarity between the reference region image feature vector and the to-be-detected region image feature vector is calculated.

[0084] Here, the cosine similarity represents the similarity between the to-be-detected region and the reference region, wherein the closer the cosine similarity is to 1, the more similar the two regions are, and vice versa, the closer the cosine similarity is to 0, the less similar the two regions are. The cosine similarity between the reference region in the reference image and the to-be-detected region in the to-be-detected image corresponding thereto is calculated to obtain the cosine similarity of the suspicious region frame.

[0085] In step S108, whether the door and window are open is determined according to the cosine similarity, and door and window opening and closing information is generated.

[0086] Optionally, as a preferred example of the present invention, the step of determining whether a door or window is open based on the cosine similarity and generating door / window opening / closing information includes:

[0087] In step S501, the minimum similarity is initialized to 1.

[0088] In step S502, each suspicious region box in the binary image of the foreground target is traversed, and the cosine similarity of the suspicious region box is compared with the minimum similarity.

[0089] In step S503, if the cosine similarity of the suspected region box is less than the minimum similarity, then the minimum similarity is updated to the cosine similarity of the suspected region box. The minimum similarity is obtained after traversing all suspected region boxes in the foreground target binary map.

[0090] In step S504, the minimum similarity is compared with a preset similarity threshold.

[0091] Here, the preset similarity threshold is preferably 0.4.

[0092] In step S505, if the minimum similarity is less than a preset similarity threshold, then the doors and windows in the image to be detected are in an open state; otherwise, the doors and windows in the image to be detected are in a closed state.

[0093] Here, in this embodiment, the similarity score of all suspicious region boxes is calculated, and then the minimum value is taken. If the minimum value is smaller than 0.4, it is determined that the image to be detected has an open door or window; otherwise, it is determined that the doors and windows in the image are closed. If the doors and windows in the image to be detected are closed, the Gaussian Mixture Model (GMM) is updated using the aligned image to increase its robustness to illumination.

[0094] In summary, this embodiment uses an inspection robot to detect door and window openings, breaking away from the existing technology that uses fixed cameras, effectively reducing the overall system cost and greatly improving reusability. When detecting door and window openings, the robustness of using auxiliary signs is low due to the influence of lighting and vibration in the scene. Therefore, this embodiment uses a neural network alignment calibration method to align images and uses deep neural network feature comparison to replace image differential comparison, enabling door and window opening detection without changing the appearance of the doors and windows. This greatly improves the accuracy and robustness of door and window detection and effectively overcomes the influence of lighting and vibration in the scene.

[0095] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0096] In an embodiment, the present application also provides a robot patrol-based door and window switch detection device, which corresponds to the robot patrol-based door and window switch detection method in the above embodiment. As shown in the figure, the robot patrol-based door and window switch detection device comprises an image acquisition module 21, an alignment module 22, a difference module 23, an image processing module 24, a mapping module 25, a feature extraction module 26, a similarity calculation module 27, and a generation module 28. The detailed description of each functional module is as follows: Figure 2

[0097] The image acquisition module 21 is used to acquire the reference image of the detection point and the to-be-detected image of the detection point based on the patrol robot;

[0098] The alignment module 22 is used to align the to-be-detected image of the detection point with the reference image of the detection point by using the SuperPoint and SuperGlue algorithms to obtain the aligned to-be-detected image of the detection point;

[0099] The difference module 23 is used to input the reference image and the aligned to-be-detected image of the detection point into the Gaussian Mixture Model (GMM) for background difference to obtain a foreground target binary image;

[0100] The image processing module 24 is used to perform image processing on the foreground target binary image to obtain a suspicious area frame;

[0101] The mapping module 25 is used to map the suspicious area frame to the reference image to obtain a reference area, and map the suspicious area frame to the aligned to-be-detected image of the detection point to obtain a to-be-detected area;

[0102] The feature extraction module 26 is used to extract features of the reference area by using a deep neural network to obtain a reference area image feature vector, and extract features of the to-be-detected area by using a deep neural network to obtain a to-be-detected area image feature vector;

[0103] The similarity calculation module 27 is used to calculate the cosine similarity between the reference area image feature vector and the to-be-detected area image feature vector;

[0104] The generation module 28 is used to determine whether the door and window are open according to the cosine similarity, and generate door and window switch information.

[0105] ​The specific limitations of the door and window switch detection device based on robot patrol can be referred to the limitations of the door and window switch detection method based on robot patrol, which will not be repeated here. Each module in the above door and window switch detection device based on robot patrol can be realized by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in software form, so that the processor calls and executes the operations corresponding to each module.

[0106] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram thereof can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a door and window switch detection method based on robot patrol.

[0107] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0108] Based on the patrol robot, a reference image of a detection point and a to-be-detected image of the detection point are collected;

[0109] The to-be-detected image of the detection point and the reference image of the detection point are aligned using a super point SuperPoint and a powerful glue SuperGlue algorithm, to obtain an aligned to-be-detected image of the detection point;

[0110] The reference image and the aligned to-be-detected image of the detection point are input into a Gaussian mixture model GMM for background difference, to obtain a foreground target binary image;

[0111] The foreground target binary image is subjected to image processing to obtain a suspicious region frame;

[0112] The suspicious region frame is mapped to the reference image to obtain a reference region, and the suspicious region frame is mapped to the aligned to-be-detected image to obtain a to-be-detected region;

[0113] extracting features of the reference region by using a deep neural network to obtain a reference region image feature vector, and extracting features of the to-be-detected region by using a deep neural network to obtain a to-be-detected region image feature vector;

[0114] calculating a cosine similarity between the reference region image feature vector and the to-be-detected region image feature vector;

[0115] judging whether the door and window are opened or not according to the cosine similarity to generate door and window opening and closing information.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-mentioned functions.

[0118] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for detecting the opening and closing of doors and windows based on robot inspection, characterized in that, include: Based on the reference image and the image to be detected at the detection point collected by the patrol robot, the reference image refers to the image of the door and window in a pre-set closed state, and the image to be detected refers to the image collected by the robot at a fixed shooting point during the inspection. The reference image is a closed image of the door and window without any abnormal interference. Using the SuperPoint and SuperGlue algorithms, the image to be detected at each detection point is aligned with a reference image at the same detection point to obtain the aligned image to be detected. Specifically, this involves: inputting the reference image of the detection point into a first SuperPoint model to extract feature point information; inputting the image to be detected into a second SuperPoint model to extract feature point information; inputting the feature point information of the reference image and the image to be detected into a SuperGlue model to generate feature matching information between the reference image and the image to be detected; and aligning the image to be detected onto the reference image based on the feature matching information to obtain the aligned image to be detected. The reference image and the image to be detected after the detection points are aligned are input into a Gaussian Mixture Model (GMM) for background subtraction to obtain a binary image of the foreground target. The number of Gaussian kernels in the GMM is 5, and the background update operation of the GMM is turned off in the subtraction calculation. Image processing is performed on the foreground target binary image to obtain suspicious region boxes. Specifically, this includes: performing morphological filtering on the foreground target binary image to obtain a filtered foreground target binary image; performing hole-filling on the filtered foreground target binary image to obtain a hole-filled foreground target binary image; performing contour extraction on the hole-filled foreground target binary image to obtain suspicious target regions in the foreground target binary image; and performing a maximum bounding box operation on the suspicious target regions in the foreground target binary image to obtain suspicious region boxes. Each foreground target binary image includes at least one suspicious region box. The suspicious region box is mapped onto the reference image to obtain the reference region, and the suspicious region box is mapped onto the image to be detected after the detection points are aligned to obtain the region to be detected. A deep neural network is used to extract features from the reference region to obtain a feature vector of the reference region image. A deep neural network is also used to extract features from the region to be detected to obtain a feature vector of the region to be detected image. The deep neural network is trained using unsupervised contrastive learning, and the loss function is CircleLoss. The training dataset used in the training process is the open-source network dataset ImageNet, which is preprocessed. Calculate the cosine similarity between the feature vector of the reference region image and the feature vector of the region to be detected image; Determining whether doors and windows are open based on the cosine similarity and generating door and window opening / closing information specifically includes: initializing the minimum similarity to 1; traversing each suspicious region box in the foreground target binary image and comparing the cosine similarity of the suspicious region box with the minimum similarity; if the cosine similarity of the suspicious region box is less than the minimum similarity, then updating the minimum similarity to the cosine similarity of the suspicious region box; until all suspicious region boxes in the foreground target binary image have been traversed, comparing the minimum similarity with a preset similarity threshold; if the minimum similarity is less than the preset similarity threshold, then the doors and windows in the image to be detected are in an open state; otherwise, the doors and windows in the image to be detected are in a closed state. The preset similarity threshold is 0.

4.

2. A door and window opening / closing detection device based on robot inspection, characterized in that, The device includes: The image acquisition module is used to collect reference images and images to be detected at the detection points based on the patrol robot. The reference image refers to a pre-set image of the closed state of doors and windows, and the image to be detected refers to an image collected by the robot at a fixed shooting point during the inspection. The reference image is a closed image of doors and windows without any abnormal interference. The alignment module employs a SuperPoint and SuperGlue algorithm to align the image to be detected at each detection point with a reference image at the same detection point, resulting in an aligned image to be detected. Specifically, this includes: inputting the reference image of the detection point into a first SuperPoint model to extract feature point information; inputting the image to be detected into a second SuperPoint model to extract feature point information; inputting the feature point information of the reference image and the image to be detected into a SuperGlue model to generate feature matching information between the reference image and the image to be detected; and aligning the image to be detected onto the reference image based on the feature matching information to obtain the aligned image to be detected. The difference module is used to input the reference image and the image to be detected after the detection points are aligned into the Gaussian Mixture Model (GMM) for background difference to obtain a binary image of the foreground target. The number of Gaussian kernels in the Gaussian Mixture Model (GMM) is 5, and the background update operation of the Gaussian Mixture Model (GMM) is turned off in the difference calculation. The image processing module is used to perform image processing on the foreground target binary image to obtain suspicious region boxes. Specifically, it includes: performing morphological filtering on the foreground target binary image to obtain a filtered foreground target binary image; performing hole-filling on the filtered foreground target binary image to obtain a hole-filled foreground target binary image; performing contour extraction on the hole-filled foreground target binary image to obtain suspicious target regions of the foreground target binary image; and performing a maximum bounding box operation on the suspicious target regions of the foreground target binary image to obtain suspicious region boxes. A foreground target binary image includes at least one suspicious region box. The mapping module is used to map the suspicious region box to the reference image to obtain the reference region, and to map the suspicious region box to the image to be detected after the detection points are aligned to obtain the region to be detected. The feature extraction module is used to extract features from the reference region using a deep neural network to obtain a feature vector of the reference region image, and to extract features from the region to be detected using a deep neural network to obtain a feature vector of the region to be detected image. The deep neural network is trained using unsupervised contrastive learning, and the loss function is CircleLoss. The training dataset used in the training process is the open-source network dataset ImageNet, and it is preprocessed. The similarity calculation module is used to calculate the cosine similarity between the feature vector of the reference region image and the feature vector of the image to be detected; The generation module is used to determine whether doors and windows are open based on the cosine similarity and generate door and window opening / closing information. Specifically, it includes: initializing the minimum similarity to 1; traversing each suspicious region box in the foreground target binary image and comparing the cosine similarity of the suspicious region box with the minimum similarity; if the cosine similarity of the suspicious region box is less than the minimum similarity, then updating the minimum similarity to the cosine similarity of the suspicious region box; until all suspicious region boxes in the foreground target binary image have been traversed, comparing the minimum similarity with a preset similarity threshold; if the minimum similarity is less than the preset similarity threshold, then the doors and windows in the image to be detected are in an open state; otherwise, the doors and windows in the image to be detected are in a closed state. The preset similarity threshold is 0.

4.

3. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the door and window opening and closing detection method based on robot inspection as described in claim 1.

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image door and window opening and closing detection method based on robot inspection as described in claim 1.

Images