Information room AR target detection system and method based on feature library matching

Abstract

The present invention discloses an AR target detection system and method for an information room based on feature library matching. The system includes establishing a three-dimensional scene of the information room, establishing device labels for the information room, obtaining the number, type, and relative position information of each device in the field of view of AR glasses in the information room scene; correlating the pixel position coordinates of each identified device in the AR glasses camera coordinate system with the coordinates of the corresponding device in the three-dimensional scene based on the relative position information of each device in the field of view of the AR glasses to obtain corresponding matching points; solving the position of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system based on the matching points, thereby obtaining the coordinates of the AR glasses in the three-dimensional scene of the computer room; and inferring the pixel positions of unidentified devices in the field of view of the AR glasses. The present invention can improve the accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of AR (augmented reality) inspection in information computer rooms, and specifically to an AR target detection system and method for information computer rooms based on feature library matching. Background Art

[0002] Since the AlexNet proposed by Hinton's team in 2012, which made significant progress in image classification, deep convolutional neural networks (DCNNs) have been widely used in the field of object detection. Scholars at home and abroad have successively proposed two-stage object detection frameworks, such as the R-CNN (region with CNN features) series, and end-to-end single-stage object detection frameworks, such as the YOLO (you only look once) series. The YOLO series is widely used in industrial scenarios for its speed and accuracy. However, as the number of network layers increases, the receptive field gradually increases. Due to the low resolution and small pixel ratio of small objects, it is difficult to extract effective feature information from them, resulting in low small object detection accuracy. Based on this, relevant scholars have proposed a series of small target detection methods. In response to the complex environmental problems in field power inspection scenarios, Single Shot Multi-Box Detector (SSD) is used for multi-scale target detection, effectively overcoming the influence of illumination and scale changes. In response to the low efficiency of traditional manual inspection methods and the proneness to missed detection and false detection, relevant scholars have proposed a multi-scale target detection algorithm based on the self-attention mechanism, which enhances the feature extraction capability of YOLOv3 in the focus area and improves the accuracy of small target detection. In response to the problem that small-area fireworks are difficult to identify, relevant scholars have proposed a target fireworks detection method based on improved YOLOX to achieve effective identification of target fireworks. The above studies all improve target detection accuracy by improving feature extraction methods. Although the target detection accuracy is improved in the scenarios where they are located, it is difficult to meet the needs of complex power grid inspections for real-time and accurate target detection. Summary of the Invention

[0003] The purpose of the present invention is to provide an information room AR target detection system and method based on feature library matching, which can accurately detect targets in real time.

[0004] To achieve this goal, the information room AR target detection system designed by the present invention based on feature library matching includes a three-dimensional scene establishment module, a device label establishment module, a device identification module, a label matching module, an AR glasses posture calculation module and an unidentified device position acquisition module;

[0005] The three-dimensional scene establishment module is used to establish a three-dimensional scene of the information room and determine the coordinates of each device in the room in the three-dimensional world;

[0006] The equipment label establishment module is used to establish equipment labels for the information room;

[0007] The device identification module is used to obtain the number, type, and relative position information of each device in the field of view of the AR glasses in the information room scenario;

[0008] The tag matching module is used to match the number, type, and relative position information of each device in the field of view of the AR glasses with the device tags in the information room, and obtain the device numbers of the identified devices in the field of view of the AR glasses, the coordinates of the identified devices in the field of view of the AR glasses in the three-dimensional scene, and the numbers of unidentified devices in the field of view of the AR glasses and the coordinates of unidentified devices in the field of view of the AR glasses in the three-dimensional scene;

[0009] The AR glasses pose calculation module is used to associate the pixel position coordinates of each identified device in the AR glasses camera coordinate system with the coordinates of the corresponding device in the three-dimensional scene based on the relative position information of each device in the AR glasses' field of view to obtain corresponding matching points. Based on the matching points, the pose of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system is solved to obtain the coordinates of the AR glasses in the three-dimensional scene of the computer room;

[0010] The unidentified device position acquisition module is used to reversely infer the pixel position of the unidentified device in the field of view of the AR glasses based on the coordinates of the AR glasses in the three-dimensional scene of the computer room, the unidentified device number in the field of view of the AR glasses, and the coordinates of the unidentified device in the three-dimensional scene in the field of view of the AR glasses.

[0011] Beneficial effects of the present invention:

[0012] The present invention performs target detection on the equipment in the computer room, and then matches them according to the labels based on the recognition results, so as to obtain devices that are not identified due to their small size. Then, according to the PnP method, the position of the AR glasses is calculated and the pixel position of the unidentified devices in the first perspective of the AR glasses is reversed. The image of the area is then enlarged and target detection is performed again, thereby improving the accuracy of small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a structural schematic diagram of the present invention;

[0014] Figure 2 This is the PnP (Perspective-n-Point) geometric structure diagram in the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0016] like Figure 1 The information room AR target detection system based on feature library matching is characterized by: including a three-dimensional scene establishment module, a device label establishment module, a device identification module, a label matching module, an AR glasses posture calculation module and an unidentified device position acquisition module;

[0017] The three-dimensional scene establishment module is used to establish a three-dimensional scene of the information room and determine the coordinates Pi (Xw, Yw, Zw) of each device in the room in the three-dimensional world;

[0018] The device tag establishment module is used to establish device tags in the information room. Device tags are used for association and matching. Each device has its own tag, which includes the device number and the device's three-dimensional coordinate information.

[0019] The device identification module is used to obtain the number, type, and relative position of each device in the field of view of the AR glasses in the information room scenario. By matching the identified devices with the tags, it can determine which unidentified devices are in the field of view.

[0020] The tag matching module is used to match the quantity, type and relative position information of each device in the field of view of the AR glasses with the device tags in the information room, and obtain the device number (N) of the identified device in the field of view of the AR glasses, the coordinates Pi (Xw, Yw, Zw) of the identified device in the field of view of the AR glasses in the three-dimensional scene, and the number (N) of the unidentified device in the field of view of the AR glasses, and the coordinates Pi (Xw, Yw, Zw) of the unidentified device in the field of view of the AR glasses in the three-dimensional scene;

[0021] The AR glasses pose calculation module is used to associate the pixel position coordinates pi (Xc, Yc) of each identified device in the AR glasses camera coordinate system with the coordinates Pi (Xw, Yw, Zw) of the corresponding identified device in the three-dimensional scene according to the relative position information of each device in the AR glasses' field of view to obtain corresponding matching points. Based on the matching points, the pose of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system is solved to obtain the coordinates of the AR glasses in the three-dimensional scene of the computer room;

[0022] The unidentified device location acquisition module is used to reversely infer the pixel position of the unidentified device in the field of view of the AR glasses based on the coordinates of the AR glasses in the three-dimensional scene of the computer room, the unidentified device number in the field of view of the AR glasses, and the coordinates of the unidentified device in the three-dimensional scene in the field of view of the AR glasses, thereby locking the approximate position of the unidentified device, which is the basis for the next step of local zoom-in re-detection.

[0023] In the above technical solution, the device identification module is used in an information room scenario. When there are devices in the computer room in the first field of view of the AR glasses, the target detection technology based on image recognition is used to perform target detection on each device in the first field of view of the AR glasses to obtain the pixel coordinates (indicating the position of the identified device in the first field of view image of the AR glasses), quantity, type and relative position information of each device in the first field of view of the AR glasses.

[0024] In the above technical solution, the device label includes the device number (N), the coordinates Pi (Xw, Yw, Zw) of each device in the three-dimensional scene, the type of device in the computer room (L) and the number of each type of device in the computer room (S).

[0025] In the above technical solution, based on the matching points associated with the pixel position coordinates of each device in the AR glasses camera coordinate system and the coordinates of the corresponding device in the three-dimensional scene, the PnP method is used based on the matching points to solve the posture of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system.

[0026] In the above technical solution, the specific method for solving the position of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system is as follows: Figure 2 As shown:

[0027] It is known that the coordinates of a device in a three-dimensional scene are: [X w 、Y w 、Z w ] T , X w 、Y w 、Z w They are the horizontal coordinate, vertical coordinate and height coordinate respectively, T represents the matrix transpose, and the homogeneous coordinate in the three-dimensional scene is [X w 、Y w 、Z w , 1] T ;

[0028] The device projects the point coordinates of [u, v] in the first-person perspective image of the AR glasses. T , T represents matrix transpose, u and v represent the horizontal and vertical coordinates of the projection point, and the homogeneous coordinates of the projection point are [u, v, 1] T ;

[0029] The camera's intrinsic parameter matrix is: K, and the camera's extrinsic parameters are: R, t;

[0030] R represents the rotation matrix from the world coordinate system to the camera coordinate system (converting the representation of the same vector in the world coordinate system to the representation in the camera coordinate system); t represents the corresponding translation vector (i.e., the vector from the origin of the camera coordinate system to the origin of the world coordinate system, represented in the camera coordinate system);

[0031] The perspective projection model is:

[0032]

[0033] Where Zc represents the depth of the device's projection point relative to the imaging plane in the first-person perspective image of the AR glasses;

[0034] Expand:

[0035]

[0036] Among them, f11, f12, f13, f14, f21, f22, f23, f24, f31, f32, f33, and f34 respectively represent the variables to be determined;

[0037] Written in the form of a system of equations:

[0038] z c u c =f 11 X w +f 12 Y w +f 13 Z w +f 14

[0039] z c v c =f 21 X w +f 22 Y w +f 23 Z w +f 24

[0040] z c =f 31 X w +f 32 Y w +f 33 Z w +f 34

[0041] Eliminate Z using the elimination method c , sorted out:

[0042] f 11 X w +f 12 Y w +f 13 Z w +f 14 -f 31 X w uf 32 Y w u c -f33 Z w u c -f 34 u c =0

[0043] f 21 X w +f 22 Y w +f 23 Z w +f 24 -f 31 X w vf 32 Y w v c -f 33 Z w v c -f 31 v c =0

[0044] The matching points corresponding to the pixel position coordinates of each device in the AR glasses camera coordinate system and the coordinates of the corresponding device in the three-dimensional scene correspond to two equations. There are a total of 12 unknowns, and at least 6 sets of matching points are required. If there are N sets of matching points, then there is a linear equation system:

[0045]

[0046] Among them, X1, Y1, and Z1 represent the horizontal coordinate, vertical coordinate, and height coordinate in the spatial coordinate system of the first group of known matching points. N 、Y N 、Z N They represent the horizontal coordinate, vertical coordinate and height coordinate in the spatial coordinate system of the Nth group of known matching points, u1 and v1 represent the horizontal and vertical coordinates in the pixel plane of the first group of known matching points, u N and v N Respectively represent the horizontal and vertical coordinates of the pixel plane of the Nth group of known matching points;

[0047] The above formula is written in matrix form:

[0048] AF=0;

[0049]

[0050]

[0051] When N≤6, the linear equations are solved directly; when N>6, the SVD (singular value decomposition) method is used to find the least squares solution of the overdetermined equations;

[0052] Therefore, the rotation matrix and translation matrix are obtained:

[0053]

[0054] in, represents the rotation matrix, represents the translation matrix, K -1 represents the inverse matrix of the camera intrinsic parameter matrix, and That is, it represents the position of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system.

[0055] In the above technical solution, the specific method for the unidentified device position acquisition module to reversely infer the pixel position of the unidentified device in the field of view of the AR glasses based on the coordinates of the AR glasses in the three-dimensional scene of the computer room, the unidentified device number in the field of view of the AR glasses, and the coordinates of the unidentified device in the three-dimensional scene in the field of view of the AR glasses is: through the above-mentioned PNP algorithm, the camera intrinsic parameter matrix K is known to be K, and the camera's extrinsic parameters R and t are solved. The coordinates Pi of the device in the three-dimensional scene can be obtained, and the pixel position coordinates pi of the device in the AR first-person perspective image can be obtained.

[0056] An information room AR target detection system based on feature library matching includes the following steps:

[0057] Step 1: Create a 3D scene of the information room and determine the coordinates Pi (Xw, Yw, Zw) of each device in the room in the 3D world;

[0058] Step 2: Create equipment labels for the information room;

[0059] Step 3: Obtain the number, type, and relative position information of each device in the field of view of the AR glasses in the information room scene;

[0060] Step 4: Match the number, type, and relative position information of each device in the AR glasses' field of view with the device labels in the information room to obtain the device number (N) of the identified device in the AR glasses' field of view, the coordinates Pi (Xw, Yw, Zw) of the identified device in the 3D scene in the AR glasses' field of view, and the number (N) of the unidentified device in the AR glasses' field of view and the coordinates Pi (Xw, Yw, Zw) of the unidentified device in the AR glasses' field of view in the 3D scene.

[0061] Step 5: Based on the relative position information of each device in the field of view of the AR glasses, the pixel position coordinates pi (Xc, Yc) of each identified device in the AR glasses camera coordinate system are associated with the coordinates of the corresponding identified device in the 3D scene to obtain the corresponding matching points. Based on the matching points, the position and posture of the AR glasses camera coordinate system relative to the 3D world coordinate system are solved to obtain the coordinates of the AR glasses in the 3D scene of the computer room;

[0062] Step 6: Reversely infer the pixel position of the unidentified device in the AR glasses' field of view based on the coordinates of the AR glasses in the 3D scene of the computer room, the number of the unidentified device in the AR glasses' field of view, and the coordinates of the unidentified device in the 3D scene in the AR glasses' field of view;

[0063] Step 7: After obtaining the pixel position of the unidentified device in the field of view, zoom in on the image of that area and perform target detection again according to steps 3 to 6 to improve the accuracy of small target detection.

[0064] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0065] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. An AR target detection system for an information room based on feature library matching, characterized by: It includes a 3D scene establishment module, a device tag establishment module, a device identification module, a tag matching module, an AR glasses posture calculation module, and an unidentified device position acquisition module; The three-dimensional scene establishment module is used to establish a three-dimensional scene of the information room and determine the coordinates of each device in the room in the three-dimensional world; The equipment label establishment module is used to establish equipment labels for the information room; The device identification module is used to obtain the number, type, and relative position information of each device in the field of view of the AR glasses in the information room scenario; The tag matching module is used to match the number, type, and relative position information of each device in the field of view of the AR glasses with the device tags in the information room, and obtain the device numbers of the identified devices in the field of view of the AR glasses, the coordinates of the identified devices in the field of view of the AR glasses in the three-dimensional scene, and the numbers of unidentified devices in the field of view of the AR glasses and the coordinates of unidentified devices in the field of view of the AR glasses in the three-dimensional scene; The AR glasses pose calculation module is used to associate the pixel position coordinates of each identified device in the AR glasses camera coordinate system with the coordinates of the corresponding identified device in the three-dimensional scene based on the relative position information of each device in the AR glasses' field of view to obtain corresponding matching points. Based on the matching points, the pose of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system is solved to obtain the coordinates of the AR glasses in the three-dimensional scene of the computer room; The unidentified device position acquisition module is used to reversely infer the pixel position of the unidentified device in the field of view of the AR glasses based on the coordinates of the AR glasses in the three-dimensional scene of the computer room, the number of the unidentified device in the field of view of the AR glasses, and the coordinates of the unidentified device in the three-dimensional scene in the field of view of the AR glasses; Based on the matching points that associate the pixel position coordinates of each device in the AR glasses camera coordinate system with the coordinates of the corresponding device in the three-dimensional scene, the PnP method is used to solve the position of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system based on the matching points.

2. The information room AR target detection system based on feature library matching according to claim 1 is characterized by: The device identification module is used in an information room scenario. When there are devices in the computer room in the first field of view of the AR glasses, the device identification module uses image recognition-based target detection technology to perform target detection on each device in the first field of view of the AR glasses to obtain pixel coordinates, quantity, type and relative position information of each device in the first field of view of the AR glasses.

3. The information room AR target detection system based on feature library matching according to claim 1 is characterized by: The device tags include the device number, the coordinates of each device in the three-dimensional scene, the type of device in the computer room, and the number of each type of device in the computer room.

4. The information room AR target detection system based on feature library matching according to claim 1 is characterized in that: The specific method for solving the pose of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system is: It is known that the coordinates of a device in a three-dimensional scene are: [X w 、Y w , Z w ] T , X w 、Y w , Z w They are the horizontal coordinate, vertical coordinate and height coordinate respectively, T represents the matrix transpose, and the homogeneous coordinate in the three-dimensional scene is [X w 、Y w , Z w , 1] T ; The device projects the point coordinates of [u, v] in the first-person perspective image of the AR glasses. T , T represents matrix transpose, u and v represent the horizontal and vertical coordinates of the projection point, and the homogeneous coordinates of the projection point are [u, v, 1] T ; The camera's intrinsic parameter matrix is: K, and the camera's extrinsic parameters are: R, t; R represents the rotation matrix from the world coordinate system to the camera coordinate system; t represents the corresponding translation vector; The perspective projection model is: Among them, z c Indicates the depth of the device's projection point relative to the imaging plane in the AR glasses' first-person perspective image; Expand: Among them, f 11 、f 12 、f 13 、f 14 、f 21 、f 22 、f 23 、f 24 、f 31 、f 32 、f 33 、f 34 Respectively represent the variables to be determined; Written in the form of a system of equations: z c u c =f 11 X ω +f 12 Y ω +f 13 Z ω +f 14 z c v c =f 21 X ω +f 22 Y ω +f 23 Z ω +f 24 z c =f 31 X ω +f 32 Y ω +f 33 Z ω +f 34 Eliminate Z using the elimination method c , sorted out: f 11 X ω +f 12 Y ω +f 13 Z ω +f 14 -f 31 X ω u-f 32 Y ω u c -f 33 Z ω u c -f 34 u c =0 f 21 X ω +f 22 Y ω +f 23 Z ω +f 24 -f 31 X ω v-f 32 Y ω v c -f 33 Z ω v c -f 31 v c =0 The matching points corresponding to the pixel position coordinates of each device in the AR glasses camera coordinate system and the coordinates of the corresponding device in the three-dimensional scene correspond to two equations. There are a total of 12 unknowns, and at least 6 sets of matching points are required. If there are N sets of matching points, then there is a linear equation system: Among them, X1, Y1, and Z1 represent the horizontal coordinate, vertical coordinate, and height coordinate in the spatial coordinate system of the first group of known matching points. N 、Y N , Z N They represent the horizontal coordinate, vertical coordinate and height coordinate in the spatial coordinate system of the Nth group of known matching points, u1 and v1 represent the horizontal and vertical coordinates in the pixel plane of the first group of known matching points, u N and v N Respectively represent the horizontal and vertical coordinates of the pixel plane of the Nth group of known matching points; The above formula is written in matrix form: AF=0; When N≤6, the linear equations are solved directly; when N>6, the SVD method is used to find the least squares solution of the overdetermined equations; Therefore, the rotation matrix and translation matrix are obtained: in, represents the rotation matrix, represents the translation matrix, K -1 represents the inverse matrix of the camera intrinsic parameter matrix, and That is, it represents the position of the AR glasses camera coordinate system relative to the three-dimensional world coordinate system.

5. The information room AR target detection system based on feature library matching according to claim 4 is characterized in that: The specific method for the unidentified device position acquisition module to reversely infer the pixel position of the unidentified device in the field of view of the AR glasses based on the coordinates of the AR glasses in the three-dimensional scene of the computer room, the unidentified device number in the field of view of the AR glasses, and the coordinates of the unidentified device in the three-dimensional scene in the field of view of the AR glasses is as follows: through the above-mentioned PnP method, the camera intrinsic parameter matrix K is known to be K, and the camera's extrinsic parameters R and t are solved. The coordinates Pi of the device in the three-dimensional scene can be obtained, and the pixel position coordinates pi of the device in the AR first-person perspective image can be obtained.

6. An information room AR target detection method based on feature library matching according to the system of claim 1, characterized in that: It includes the following steps: Step 1: Create a 3D scene of the information room and determine the coordinates of each device in the room in the 3D world; Step 2: Create equipment labels for the information room; Step 3: Obtain the number, type, and relative position information of each device in the field of view of the AR glasses in the information room scene; Step 4: Match the quantity, type, and relative position information of each device in the AR glasses' field of view with the device labels in the information room to obtain the device numbers of the identified devices in the AR glasses' field of view, the coordinates of the identified devices in the 3D scene in the AR glasses' field of view, and the numbers and coordinates of the unidentified devices in the AR glasses' field of view in the 3D scene. Step 5: Based on the relative position information of each device in the field of view of the AR glasses, the pixel position coordinates of each identified device in the AR glasses camera coordinate system are associated with the coordinates of the corresponding identified device in the 3D scene to obtain corresponding matching points. Based on the matching points, the position and pose of the AR glasses camera coordinate system relative to the 3D world coordinate system are solved to obtain the coordinates of the AR glasses in the 3D scene of the computer room; Step 6: Reversely infer the pixel position of the unidentified device in the field of view of the AR glasses based on the coordinates of the AR glasses in the 3D scene of the computer room, the number of the unidentified device in the field of view of the AR glasses, and the coordinates of the unidentified device in the 3D scene in the field of view of the AR glasses.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 6 are implemented.

Images