Communication machine room equipment state detection method based on augmented reality
By applying augmented reality technology and space mapping technology in the communication room, combined with indicator light recognition algorithm, the complexity and accuracy of device status detection are solved, real-time, accurate and intuitive detection of device status is achieved, and inspection efficiency is improved .
Patent Information
- Application Number
- CN202410944946.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art has problems such as complex operation, insufficient adaptability and low recognition accuracy in the detection of equipment status of communication room, especially in complex environments, it is difficult to achieve real-time and efficient requirements.
Using augmented reality-based method, a three-dimensional environmental model is created through spatial mapping technology, combining indicator light recognition and positioning algorithm and Hoff circle detection algorithm, images are collected and processed in real time, device status is recognized, and virtual information is superimposed in the real environment to realize the device Visual display of status.
It improves user patrol efficiency, realizes real-time, accurate and intuitive detection of equipment status, simplifies the detection process, is highly adaptable and does not require manual identification.
Smart Images

Figure CN118941999A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the application fields of computer vision and augmented reality, and in particular to an augmented reality-based communication room equipment status detection method. Background Art
[0002] With the rapid development of information technology, data center rooms are responsible for the centralized processing, storage, transmission, exchange, and management of information. They are the physical space for realizing information management and can ensure the stable operation of the enterprise data platform. As an important part of key infrastructure, communication rooms are particularly important for real-time monitoring and inspection of equipment status. However, equipment status detection in a complex and changing environment is often interfered by environmental factors, resulting in reduced detection efficiency and causing certain troubles for operation and maintenance.
[0003] At present, the field of equipment status recognition generally has problems such as complex operation, insufficient adaptability, and low recognition accuracy. Auxiliary camera recognition is still one of the mainstream methods. In outdoor environments, the application of inspection robots is still relatively limited. They are easily affected by route obstacles and can only collect images but cannot directly identify them. Therefore, they still need to be equipped with corresponding recognition technology or transmit images back for manual recognition. However, manual recognition is inefficient and cannot guarantee real-time and accuracy, which cannot meet current detection requirements. In addition, some methods collect videos of the indicator light position and use the indicator light state discrimination model to identify the on and off state of the indicator light in the video frame image. However, this method uses video frame images to process frame by frame, which has high requirements for processing speed and computing resources, and may not be suitable for scenarios with high real-time requirements. How to deal with the real-time, intuitive and efficient feedback requirements of the communication room inspection process needs further exploration and improvement. Summary of the invention
[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a communication room equipment status detection method based on augmented reality, which can perform status detection on the equipment in the communication room, achieve efficient inspection feedback effect in an enhanced display environment, and realize visualization and digitization of the inspection process, which specifically includes the following steps:
[0005] Step 1: Scan the actual communication room by using spatial mapping technology, and create an environmental three-dimensional model of the actual communication room within the scanning field of view of the scanning device, wherein the environmental three-dimensional model at least includes a three-dimensional geometric model of each object within the scanning field of view and the spatial coordinates of each object within the scanning field of view;
[0006] Step 2: Analyze the three-dimensional model of the environment to obtain spatial data of the target device to be detected, wherein the spatial data at least includes the position of the target device to be detected in the scanning field of view and the direction of the target device to be detected relative to the scanning device, and collect images of the target device to be detected within the scanning field of view;
[0007] Step 3: Preprocess the collected image, and detect and identify the preprocessed image through the indicator light recognition and positioning algorithm to obtain the mark of the indicator light in the preprocessed image;
[0008] Step 4: At the position where the indicator light mark is located in the preprocessed image, the indicator light is color recognized to determine the color corresponding to the indicator light and the state of the indicator light, where the state of the indicator light is whether the indicator light is on, and the specific color and state of the indicator light are marked in the preprocessed image, and the marked image is converted to obtain a target image, and the target image is output;
[0009] Step 5: Based on the position of the target device in the scanning field of view and the direction of the target device to be detected relative to the scanning device, present the target image to the target device in the scanning field of view so that the state and color of the target device can be observed in the scanning field of view of the scanning device.
[0010] Furthermore, step 1 specifically includes:
[0011] Scanning a real communication room through a spatial mapping technology to obtain a spatial anchor point file, a spatial grid file, and a spatial mapping data file, wherein the spatial anchor point file includes the three-dimensional coordinates of each object within the scanning field of view, the spatial grid file includes the geometric shape and structure of each object within the scanning field of view, and the spatial mapping data file includes environmental feature points and depth information, wherein the environmental feature points are points within the scanning field of view that can help build a three-dimensional model of the environment so as to use an augmented reality device for spatial positioning and mapping, and the depth information is distance information detected by a depth sensor, and the distance information includes the distance between the scanning device and the surface of each object within the scanning field of view, and is used to create a three-dimensional model of the environment;
[0012] Based on the three-dimensional coordinates of each object in the scanning field of view in the spatial anchor point file, and the environmental feature points and depth information in the spatial mapping data file, a three-dimensional model of the environment of the real communication room within the scanning field of view of the scanning device is created.
[0013] Furthermore, step 3 specifically includes:
[0014] Preprocessing the collected image includes: performing type conversion on the collected image, and performing dilation convolution on the converted image to reduce interference of identification information of the indicator light on identification of the indicator light on the target device, wherein the identification information is used to characterize the meaning represented by the indicator light;
[0015] The detection and recognition of the preprocessed image includes: using the Hough circle detection and recognition algorithm to detect and recognize the indicator light on the preprocessed image to obtain the radius and coordinates of the indicator light, and in the preprocessed image, marking the indicator light by drawing a circle according to the radius and coordinates of the indicator light.
[0016] Furthermore, step 4 specifically includes:
[0017] At the position of the indicator light in the image after preprocessing, the color of the indicator light is recognized to obtain a numerical value representing the hue, a numerical value representing the saturation, and a numerical value representing the brightness; the color of the indicator light and the state of the indicator light are determined according to a pre-set threshold range of the numerical value representing the hue of different colors, a threshold range of the numerical value representing the saturation of different colors, and a threshold range of the numerical value representing the brightness of different colors.
[0018] The beneficial effects of adopting the above technical solution are:
[0019] 1. The present invention has the advantage of visualization. The device status is detected in an augmented reality environment, and the status recognition result of the target device is directly presented to the user by superimposing virtual information in the real environment through augmented reality display technology, thereby improving the user's inspection efficiency;
[0020] 2. The present invention has the advantage of being easy to use. The device status detection model can be directly called during the software development process. By simply reaching the target detection device, the image and spatial data can be collected with one click to obtain the detection result.
[0021] 3. The present invention has the advantage of being adjustable. The parameters in the detection model can be adjusted according to different environmental requirements, without the need for model training, and is more universal. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A logical schematic diagram of a communication room equipment status detection method based on augmented reality in an embodiment of the present invention;
[0023] Figure 2 4 is a diagram of the image processing conversion algorithm architecture in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0025] In view of the problems existing in the prior art, the present invention provides a communication room equipment status detection method based on augmented reality. Figure 1 , which may include the following steps:
[0026] Step 1: Scan the actual communication room by using spatial mapping technology, and create an environmental three-dimensional model of the actual communication room within the scanning field of view of the scanning device, wherein the environmental three-dimensional model at least includes a three-dimensional geometric model of each object within the scanning field of view and the spatial coordinates of each object within the scanning field of view;
[0027] Among them, the three-dimensional geometric model includes the shape and structure that reflects the physical environment, including walls, floors, ceilings and other fixed objects; the environmental three-dimensional model also includes spatial coordinate information, that is, the position of each point or object in the environmental three-dimensional model.
[0028] Specifically, the present invention scans a real communication room through a space mapping technology to obtain a space anchor point file, a space grid file and a space mapping data file.
[0029] Among them, the spatial mapping technology can be augmented reality AR (Augmented Reality) spatial mapping technology, which is used to scan the real communication room. In specific implementation, the real communication room can be scanned by wearing AR glasses. The created environmental three-dimensional model is a three-dimensional model of the real communication room within the field of view of the AR glasses. When the user adjusts the field of view while wearing AR glasses, the present invention will create a new environmental three-dimensional model in real time according to the field of view.
[0030] Among them, the spatial anchor point file includes the three-dimensional coordinates of each object within the scanning field of view, which can also be understood as the spatial anchor point file storing the anchor point information within the scanning field of view, that is, the spatial coordinates, recording the three-dimensional coordinates of the position of each object in the current scanning field of view, which is used to accurately locate the object; the spatial grid file includes the geometric shape and structure of each object within the scanning field of view, which can also be understood as the spatial grid file storing the three-dimensional grid data generated by the AR glasses scanning, describing the geometric shape and structure in the scanning field of view, the spatial grid file contains the vertices, edges and faces of the polygon, that is, the data used to construct the environmental three-dimensional model of the real communication room within the scanning field of view; the spatial mapping data file environmental feature points and depth information, the environmental feature points are points within the scanning field of view that can help build the environmental three-dimensional model, so as to use augmented reality equipment for spatial positioning and mapping, environmental feature points refer to points that are easily identifiable and unique in physical space, these points can significantly help augmented reality devices to locate and map in the environment. Specifically, environmental feature points may include: corner points: such as the junction of walls, ceilings, and floors; edge points: such as the edges of objects such as tables and door frames; points with obvious texture changes: such as decorations, patterns, and signs on walls; points with obvious light changes: such as the junction of light and shadow; these feature points can be accurately identified and tracked during the scanning and positioning process due to their uniqueness and stability, thereby helping AR devices build and maintain spatial models; the depth information is the distance information detected by the depth sensor, and the distance information includes the distance between the scanning device and the surface of each object within the current scanning field of view, which is used to create a three-dimensional model of the environment, and the three-dimensional model of the environment created based on the depth information is more accurate. Depth information refers to the information captured by a depth sensor (such as LiDAR), which is configured in the AR device.
[0031] Based on the three-dimensional coordinates, environmental feature points and depth information of each object in the scanning field of view in the spatial anchor point file, a three-dimensional model of the environment of the real communication room within the scanning field of view of the scanning device is created.
[0032] Among them, three-dimensional coordinates are used to accurately construct the geometric structure of the environment, environmental feature points are used as reference points to ensure that the model is accurately aligned with the actual environment, and depth information is used to help generate accurate three-dimensional shapes and structures.
[0033] Step 2: Analyze the three-dimensional model of the environment to obtain spatial data of the target device to be detected, wherein the spatial data at least includes the position of the target device to be detected in the scanning field of view and the direction of the target device to be detected relative to the scanning device, and collect images of the target device to be detected within the scanning field of view;
[0034] That is to say, the spatial data of the target device to be detected is obtained from the three-dimensional geometric model of each object in the scanning field of view in the three-dimensional model of the environment and the spatial coordinates of each object, that is, the position of the target device to be detected in the scanning field of view and the direction of the target device to be detected relative to the scanning device.
[0035] Among them, image acquisition of the target device to be detected within the scanning field of view is achieved through a scanning device, which is the AR glasses mentioned above.
[0036] Step 3: pre-process the collected image, and detect and identify the pre-processed image through the indicator light recognition and positioning algorithm to obtain the indicator light mark in the pre-processed image;
[0037] Specifically, the preprocessing of the collected image includes: performing type conversion on the collected image, specifically, converting the type of the collected image into a Mat type for subsequent image processing; performing dilation convolution on the type-converted image to reduce the interference of the indicator light identification information on the indicator light identification on the target device, wherein the identification information is used to characterize the meaning represented by the indicator light, and the identification information is represented in the form of an identification, which can be a symbol or a letter. For example, if the indicator light is identified as power, the indicator light represents the power supply. The interference of the identification information on the indicator light identification on the target device refers to the presence of a circular shape in the identification information, such as the letter O, or even some circular shapes and patterns that are very similar to the indicator light. This type of identification information will be recognized as an indicator light during identification and positioning, affecting the accuracy of identification, so it is necessary to reduce this interference through dilation convolution to improve the accuracy of identification.
[0038] Among them, the size of the dilated convolution kernel is 5, and the Hough circle detection adopts the gradient-based Hough circle change. The key parameters are as follows: the inverse ratio of the accumulator resolution to the image resolution is 1, the high threshold passed to the Canny edge detection is 12, the low threshold is determined by half of the high threshold, the low threshold is 6, and the center accumulator threshold is 25.
[0039] The purpose of converting the image type to the Mat type is to utilize various image processing functions in the OpenCV library, namely dilated convolution and Hough circle detection. Opencv is an open source library for real-time computer vision, and the Mat type of OpenCV is its core data structure. When converting the image type, an image processing conversion algorithm that can be applied to augmented reality environments is used to easily convert between Texture2D and Mat types in Unity.
[0040] The detection and recognition of the preprocessed image includes: using the Hough circle detection and recognition algorithm to detect and recognize the indicator light on the preprocessed image to obtain the radius and coordinates of the indicator light, and in the preprocessed image, marking the indicator light by drawing a circle according to the radius and coordinates of the indicator light.
[0041] Step 4: At the position where the indicator light mark is located in the preprocessed image, the indicator light is color recognized to determine the color corresponding to the indicator light and the state of the indicator light, where the state of the indicator light is whether the indicator light is on, and the specific color and state of the indicator light are marked in the preprocessed image, and the marked image is converted to obtain a target image, and the target image is output;
[0042] Specifically, the color of the indicator light is recognized at the position where the mark of the indicator light is located in the image after preprocessing, and a numerical value representing the hue, a numerical value representing the saturation, and a numerical value representing the brightness are obtained. The color of the indicator light and the state of the indicator light are determined according to a preset threshold range of numerical values representing the hue of different colors, a threshold range of numerical values representing the saturation of different colors, and a threshold range of numerical values representing the brightness of different colors. When the color of the indicator light is gray or black, it indicates that the state of the indicator light is not on, and when the color of the indicator light is other colors, it indicates that the state of the indicator light is on.
[0043] Among them, color recognition is based on the characteristics of the HSV (Hue, Saturation, Value) color space and the set threshold value. Hue means hue, which indicates the basic attribute of color, that is, the type of color, such as red, blue, green, etc. Saturation means saturation, which indicates the purity or intensity of color. It describes the gray content of the color, and Value means brightness, which indicates the brightness or intensity of the color, which describes the brightness of the color.
[0044] The present invention converts the obtained image into the HSV color space according to the HSV color component range in Table 1, and determines the color and state of the indicator light according to the value range of the color component.
[0045] Table 1 HSV color component range
[0046]
[0047] In combination with Table 1, in the process of color recognition of the indicator light, the present invention can identify the hue value, saturation value and brightness value of the indicator light, and then determine the color of the indicator light according to the threshold value in Table 1.
[0048] Among them, the specific color and status of the indicator light are marked in the preprocessed image, that is, Figure 1The information in the marking is as follows: specifically, the indicator light that is lit is marked with color, and the black one that is not lit is marked as OFF.
[0049] The type conversion of the annotated image is to convert the annotated image into the Texture2d type. The Texture2D type is a standard image format in AR platforms such as Unity, which is used for texture mapping and display. In Unity, all texture and image operations are basically completed through the Texture2D type. Converting the processed image to Texture2D can be used directly for rendering and display in the AR environment. Other image types can be used in specific cases, but Texture2D is the most commonly used and compatible choice in AR applications.
[0050] Among them, the image type conversion is based on the image type conversion method, specifically, combined with Figure 2 , the image type conversion method includes the following three parts:
[0051] 1. Packaging method
[0052] The tool class encapsulates the TextureToMat method and the MatToTexture method, which are used to easily convert between Texture2D and Mat objects in Unity. These methods call the underlying native methods according to the given parameters to complete the conversion, and provide default parameters to simplify use. The TextureToMat method and the MatToTexture method are custom names. The encapsulated methods allow developers to pass higher-level parameters when calling, such as whether to flip vertically or horizontally, the rotation angle, etc.
[0053] Color32Bytes is a custom structure used to convert between Unity's Color32 array and byte array. Because the underlying native methods usually require byte arrays as input, and Color32 arrays are often used in Unity to represent image data. Conversion between Color32 arrays and byte arrays usually does not involve mathematical formulas or complex algorithms. The Color32 structure represents color and contains four components: RGBA. The byte array is a sequence of bytes that can directly represent the components of the color. The conversion process actually extracts the color information from the Color32 array and then converts it to a byte array. When converting to a byte array, the value of each color component can be directly converted to a byte representation because the data representation methods of Color32 and byte are compatible. Color32Bytes is used because the underlying native methods need to receive byte arrays to represent image data, and Color32 arrays are often used in Unity.
[0054] 2. TextureConversionParams
[0055] TextureConversionParam is a custom structure. The name of TextureConversionParam is custom. TextureConversionParam is designed to provide more flexible image conversion and is used to store conversion parameters, including vertical flip, horizontal flip, and rotation angle, making the calling code clearer and easier to read.
[0056] 3. Opencv
[0057] The underlying method in the OpenCV library is called through the DllImport instruction to convert Unity's pixel data into OpenCV's Mat object, and to convert OpenCV's Mat object into Unity's Texture2D object. These methods include utils_texture_to_mat and utils_mat_to_texture_2.
[0058] Among them, DllImport is an attribute used in C# to call unmanaged code (such as DLL libraries written in C / C++). Through it, functions written in other programming languages can be used in C#. In the Unity project, the DllImport instruction can be used to call functions in the OpenCV library to implement image processing functions.
[0059] This architectural design makes image processing in Unity more convenient. Developers can easily exchange and process data between Unity and OpenCV without worrying about the underlying implementation details.
[0060] Step 5: Based on the position of the target device in the scanning field of view and the direction of the target device to be detected relative to the scanning device, present the target image to the target device in the scanning field of view so that the state and color of the target device can be observed in the scanning field of view of the scanning device.
[0061] Sensors, cameras and depth perception technology are used to capture the user's surroundings, and spatial positioning and tracking algorithms are used to accurately present the target image to the location of the target device in the scanning field of view.
[0062] That is to say, after wearing the augmented reality glasses, the user can observe the status and color of the target device in the scanning field of view of the scanning device, so that the target image can be presented in the AR scene.
[0063] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.
Claims
1. A communication room equipment status detection method based on augmented reality, characterized in that: include: Step 1: Scan the actual communication room by using spatial mapping technology, and create an environmental three-dimensional model of the actual communication room within the scanning field of view of the scanning device, wherein the environmental three-dimensional model at least includes a three-dimensional geometric model of each object within the scanning field of view and the spatial coordinates of each object within the scanning field of view; Step 2: Analyze the three-dimensional model of the environment to obtain spatial data of the target device to be detected, wherein the spatial data at least includes the position of the target device to be detected in the scanning field of view and the direction of the target device to be detected relative to the scanning device, and collect images of the target device to be detected within the scanning field of view; Step 3: Preprocess the collected image, and detect and identify the preprocessed image through the indicator light recognition and positioning algorithm to obtain the mark of the indicator light in the preprocessed image; Step 4: At the position where the indicator light mark is located in the preprocessed image, the indicator light is color recognized to determine the color corresponding to the indicator light and the state of the indicator light, where the state of the indicator light is whether the indicator light is on, and the specific color and state of the indicator light are marked in the preprocessed image, and the marked image is converted to obtain a target image, and the target image is output; Step 5: Based on the position of the target device in the scanning field of view and the direction of the target device to be detected relative to the scanning device, present the target image to the target device in the scanning field of view so that the state and color of the target device can be observed in the scanning field of view of the scanning device.
2. The method for detecting the state of equipment in a communication room based on augmented reality according to claim 1, characterized in that: Step 1 specifically includes: Scanning a real communication room through a spatial mapping technology to obtain a spatial anchor point file, a spatial grid file, and a spatial mapping data file, wherein the spatial anchor point file includes the three-dimensional coordinates of each object within the scanning field of view, the spatial grid file includes the geometric shape and structure of each object within the scanning field of view, and the spatial mapping data file includes environmental feature points and depth information, wherein the environmental feature points are points within the scanning field of view that can help build a three-dimensional model of the environment so as to use an augmented reality device for spatial positioning and mapping, and the depth information is distance information detected by a depth sensor, and the distance information includes the distance between the scanning device and the surface of each object within the scanning field of view, and is used to create a three-dimensional model of the environment; Based on the three-dimensional coordinates of each object in the scanning field of view in the spatial anchor point file, and the environmental feature points and depth information in the spatial mapping data file, a three-dimensional model of the environment of the real communication room within the scanning field of view of the scanning device is created.
3. The method for detecting the state of equipment in a communication room based on augmented reality according to claim 1, characterized in that: Step 3 specifically includes: Preprocessing the collected image includes: performing type conversion on the collected image, and performing dilation convolution on the converted image to reduce interference of identification information of the indicator light on identification of the indicator light on the target device, wherein the identification information is used to characterize the meaning represented by the indicator light; The detection and recognition of the preprocessed image includes: using the Hough circle detection and recognition algorithm to detect and recognize the indicator light on the preprocessed image to obtain the radius and coordinates of the indicator light, and in the preprocessed image, marking the indicator light by drawing a circle according to the radius and coordinates of the indicator light.
4. The method for detecting the state of equipment in a communication room based on augmented reality according to claim 1, characterized in that: Step 4 specifically includes: At the position of the indicator light mark in the image after preprocessing, the color of the indicator light is recognized to obtain a numerical value representing the hue, a numerical value representing the saturation, and a numerical value representing the brightness; the color of the indicator light and the state of the indicator light are determined according to a pre-set threshold range of the numerical value representing the hue of different colors, a threshold range of the numerical value representing the saturation of different colors, and a threshold range of the numerical value representing the brightness of different colors.
Citation Information
Patent Citations
Machine room inspection method based on image processing technology
CN110796754A
Machine room equipment online identification method based on image identification technology
CN110956086A
Equipment indicating lamp identification method and system based on YOLOv4
CN112560816A
Augmented reality processing method, storage medium, and electronic device
US20220392104A1
