Multi-target identification detection method and system based on UWB and machine vision fusion

Abstract

The invention relates to a multi-target identification detection method and system based on UWB and machine vision fusion. The method comprises the following steps: firstly, acquiring three-dimensional coordinate data of a monitored object and a camera, and acquiring Euler angle parameter information of the camera; according to the method, three-dimensional coordinate data are utilized to obtain camera coordinate data through relative position vector conversion, Euler angle parameter information is utilized to obtain two-dimensional plane coordinate data through conversion, normalization processing is carried out to obtain pixel coordinate data of a monitored object, a camera is utilized to collect the monitored object, an image obtained through collection is processed, and the monitored object is obtained. Obtaining visual measurement data; carrying out fusion processing on the vision measurement data and the confidence coefficient complementary value to obtain a confidence coefficient result; and finally, according to a confidence degree result, marking out a target detection mark box of which the confidence degree is greater than a preset confidence degree to complete multi-target identification. Compared with the prior art, the method has the advantages that the confidence is improved, the target detection precision is improved, the detection adaptability is high, and the reliability is high.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target recognition and detection, and in particular to a multi-target recognition and detection method and system based on the fusion of UWB and machine vision. Background Art

[0002] Currently, as an important achievement of the combination of automation and intelligence, machine vision technology is gradually becoming the new standard in the field of industrial inspection. However, for traditional machine vision systems, when dealing with out-of-focus detected objects, the appearance of multi-targets or group targets, and the situation of being occluded in complex scenes, the accuracy of target detection is challenged. Although machine vision algorithms have undergone multiple iterations, they still face the above challenges. Especially in dynamic scenes, traditional visual recognition algorithms often cause problems such as feature loss and low recognition confidence, with low confidence and inaccurate recognized targets.

[0003] UWB technology uses nanosecond-level non-sinusoidal narrow pulses to transmit data, and has advantages such as ultra-wideband and strong anti-interference ability. It is particularly suitable for high-speed wireless access and high-precision positioning in dense multipath places such as indoors, ensuring that the position information of the monitored object can be stably provided even under complex and changeable layouts and electromagnetic interference. However, existing systems that integrate UWB technology and machine vision algorithms cannot achieve data synchronization. How to ensure their efficient synchronization and avoid errors caused by time differences is the key to ensuring detection accuracy. At the same time, under the influence of the complexity and dynamic changes in the current industrial environment, existing fusion systems lack a certain degree of adaptability and still cannot maintain high accuracy and robustness in complex situations such as occlusion, out-of-focus, and multi-target overlap. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide a multi-target recognition and detection method and system based on the fusion of UWB and machine vision.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to one aspect of the present invention, a multi-target recognition and detection method based on the fusion of UWB and machine vision is provided. In this method, UWB positioning base stations are evenly arranged within a preset monitored scene range, and a camera is set to perform target recognition on the monitored object. Both the monitored object and the camera carry UWB tags. The method steps include:

[0007] S1. Collect three-dimensional coordinate data of the monitored object and the camera, and obtain the Euler angle parameter information of the camera;

[0008] S2. Calculate the pixel coordinate data of the monitored object based on the three-dimensional coordinate data of the monitored object and the camera, combined with the Euler angle parameter information;

[0009] S3. Use a camera to collect the monitored object and process the collected image to obtain visual measurement data;

[0010] S4. Calculate a confidence result from the visual measurement data and the pixel coordinate data; S5. According to the confidence result, mark the target detection bounding boxes where the confidence is greater than the preset confidence to complete multi-target recognition.

[0011] As a preferred technical solution, the coordinate system adopted for the three-dimensional coordinate data in S1 is a three-dimensional space coordinate system established with the main base station in the UWB positioning base station as the fixed reference point. The specific coordinates of the three-dimensional coordinate data of the monitored object and the camera collected are:

[0012] P_goal = (x, y, z) = (xg, yg, zg)

[0013] P_camera = (x', y', z') = (xc, yc, zc)

[0014] Among them, P_goal is the three-dimensional coordinate data of the monitored object; P_camera is the three-dimensional coordinate data of the camera; x, y, and z are the coordinate axis directions; xg, yg, and zg are the three coordinate axis variables of the monitored object in the three-dimensional coordinate system established by the UWB base station; xc, yc, and zc are the three coordinate axis variables of the camera in the three-dimensional coordinate system established by the UWB base station.

[0015] As a preferred technical solution, the specific process of obtaining the Euler angle parameter information of the camera in S1 is: Set the initial orientation of the camera to the positive Z-axis (0, 0, 1) in the established three-dimensional space coordinate system, collect the rotation matrix R of the camera, and multiply the initial camera orientation vector (0, 0, 1) by the rotation matrix to obtain the Euler angle parameter information of the camera.

[0016] As a preferred technical solution, the steps to obtain the pixel coordinate data of the monitored object in S2 are: First, use the three-dimensional coordinate data of the camera and the monitored object to find the relative position vector, then use this relative position vector to convert the three-dimensional coordinate data of the monitored object into camera coordinate data, and then use the Euler angle parameter information of the camera to convert the camera coordinate data of the monitored object into two-dimensional plane coordinate data, and finally normalize the two-dimensional plane coordinate data of the monitored object to obtain the pixel coordinate data of the monitored object.

[0017] As a preferred technical solution, the specific formula for converting the three-dimensional coordinate data of the monitored object into camera coordinate data is:

[0018]

[0019] Among them, goal_camera is the camera coordinate data; P_relactive is the relative position vector of the camera; R is the rotation matrix of the camera; x, y, and z are the coordinate axis directions; xg, yg, and zg are the three coordinate axis variables of the monitored object in the three-dimensional coordinate system established by the UWB base station; xc, yc, and zc are the three coordinate axis variables of the camera in the three-dimensional coordinate system established by the UWB base station; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data of each direction of the relative position vector.

[0020] As a preferred technical solution, the specific formula for converting the camera coordinate data of the monitored object into two-dimensional plane coordinate data is:

[0021]

[0022] Among them, goal_2Dsurface is the two-dimensional plane coordinate data; k is the parameter matrix of the image plane where the monitored object is collected by the camera; goal_camera is the camera coordinate data; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data of each direction of the relative position vector; x and y are the coordinate axis directions; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described in pixels; hy is the focal length in the y-axis direction described in pixels; cx and cy are the coordinates of the optical center corresponding to the principal optical axis of the camera in the pixel coordinate system.

[0023] As a preferred technical solution, the specific formula for normalizing the two-dimensional plane coordinate data of the monitored object is:

[0024] goal_pixel = (u_final, v_final)

[0025]

[0026] Among them, goal_pixel is the pixel coordinate data, u_final and v_final are the coordinate representations of the monitored object in the pixel coordinate system; x and y are the coordinate axis directions; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data of each direction of the relative position vector; R is the rotation matrix of the camera; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described in pixels; hy is the focal length in the y-axis direction described in pixels; cx and cy are the coordinates of the optical center corresponding to the principal optical axis of the camera in the pixel coordinate system.

[0027] As a preferred technical solution, the specific steps of S4 are as follows: First, map the pixel coordinate data to obtain the confidence complementary value of the corresponding block, and then fuse the confidence complementary value with the visual measurement data to obtain the confidence result. The specific formula for the confidence result is:

[0028] confidence_final = A·vision_confidence+(1 - A)·goal_confidence

[0029] Where confidence_final is the confidence result; vision_confidence is the visual measurement data; goal_confidence is the confidence complementary value, and A is the complementary filtering factor.

[0030] As a preferred technical solution, the preset confidence in S4 is 1 - A.

[0031] According to another aspect of the present invention, a multi-target recognition and detection system based on the fusion of UWB and machine vision is provided. This system operates using a multi-target recognition and detection method based on the fusion of UWB and machine vision as described above. The system includes a data acquisition module, a data processing module, and a target recognition module;

[0032] Among them, the data acquisition module includes a camera, a gyroscope, a UWB positioning tag, and a UWB positioning base station. This module is used to collect the three-dimensional coordinate data of the monitored object and the camera, and obtain the Euler angle parameter information of the camera;

[0033] The data processing module is used to calculate the pixel coordinate data of the monitored object based on the three-dimensional coordinate data of the monitored object and the camera, combined with the Euler angle parameter information; it is also used to process the image collected by the camera to obtain visual measurement data;

[0034] The target recognition module calculates the confidence result from the visual measurement data and the pixel coordinate data; and then, according to the confidence result, marks the target detection marking frame where the confidence is greater than the preset confidence to complete multi-target recognition.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. In the present invention, first, the three-dimensional coordinate data of the object under surveillance and the camera are collected, and the Euler angle parameter information of the camera is obtained. Then, based on the three-dimensional coordinate data of the object under surveillance and the camera, combined with the Euler angle parameter information, the pixel coordinate data of the object under surveillance are calculated. Then, the camera is used to collect the object under surveillance, and the collected image is processed to obtain visual measurement data. And from the visual measurement data and the pixel coordinate data, a confidence result is calculated. Finally, according to the confidence result, a target detection marking frame with a confidence greater than the preset confidence is marked to complete multi-target recognition. This method addresses the problem of feature loss and low recognition confidence of traditional visual recognition algorithms caused by the object being outside the camera focal plane (defocused or blurred). By complementing the pixel coordinates obtained by UWB with the confidence of the machine vision recognition module, the confidence is effectively improved, making target detection more efficient and significantly reducing the loss of objects under surveillance in the picture.

[0037] 2. In the present invention, the camera is set to perform target recognition on the object under surveillance. Both the object under surveillance and the camera carry UWB tags, and the coordinate system for the three-dimensional coordinate data is a three-dimensional space coordinate system established with the main base station in the UWB positioning base station as the fixed reference point. This enables, in the case where the target within the camera's field of view is physically blocked (such as by a wall, etc.), the full utilization of the characteristics of the UWB technology. Even in the presence of an obstacle, the corresponding focal length can be obtained and the pixel coordinates of the target object can be calculated, while marking the occluded object to achieve a perspective effect, expanding the application scenario of target detection. At the same time, by introducing the UWB technology, it is possible to achieve correction on the basis of traditional vision, significantly improving the accuracy of target recognition. Especially in complex scenarios, more reliable target detection is achieved.

[0038] 3. In the present invention, the steps to obtain the pixel coordinate data of the object under surveillance are as follows: First, using the three-dimensional coordinate data of the camera and the object under surveillance, the relative position vector of the camera is obtained. Then, using this relative position vector, the three-dimensional coordinate data of the surveillance object are converted into camera coordinate data. Then, using the Euler angle parameter information of the camera, the camera coordinate data of the surveillance object are converted into two-dimensional plane coordinate data. Finally, the two-dimensional plane coordinate data of the object under surveillance are normalized to obtain the pixel coordinate data of the object under surveillance. By combining machine vision technology and using the Euler angle parameter information to finally convert the three-dimensional coordinate data into pixel coordinate data, this method can accurately identify targets in complex environments and can be flexibly applied in various application scenarios, with strong adaptability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the steps of the multi-target recognition and detection method based on the fusion of UWB and machine vision in the present invention;

[0040] Figure 2 It is the architecture diagram of the multi - target recognition and detection system based on the fusion of UWB and machine vision in the embodiment;

[0041] Figure 3 It is the flowchart of the multi - target recognition and detection method based on the fusion of UWB and machine vision in the embodiment;

[0042] Figure 4 It is the schematic diagram of the three - dimensional data acquisition coordinate space constructed in the embodiment;

[0043] Figure 5 It is the schematic diagram of converting the camera coordinates into two - dimensional image plane coordinates in the embodiment;

[0044] Figure 6 It is the schematic diagram of the fusion of the gyroscope module and coordinate data in the embodiment. Specific implementation manners

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] As an important achievement of the combination of automation and intelligence, machine vision technology is gradually becoming the new standard in the field of industrial inspection. However, the accuracy and efficiency of target detection in traditional machine vision systems are challenged in the cases of defocused detected objects, the appearance of multi - targets or group targets, and being occluded in complex scenes. Currently, although the fusion of UWB and machine vision can greatly improve the accuracy and stability of target detection, in order to overcome these technical difficulties, in - depth research and innovation are still needed in aspects such as hardware, algorithms, and real - time processing.

[0047] To overcome these problems, an innovative detection method is provided in this application based on the fusion of UWB technology and machine vision. The UWB technology uses nanosecond-level non-sinusoidal narrow pulses to transmit data, with advantages such as ultra-wideband and strong anti-interference ability. It is especially suitable for high-speed wireless access and high-precision positioning in dense multipath scenarios such as indoors, ensuring that the position information of the monitored object can be stably provided even under complex and changing layouts and electromagnetic interference. Aiming at the problems of feature loss and low recognition confidence of traditional visual recognition algorithms caused by the object being outside the camera focal plane (defocused or blurred), by complementing the pixel coordinates obtained by UWB with the confidence of the machine vision recognition module, the confidence is effectively improved, making the target detection more efficient and significantly reducing the loss of monitored objects in the picture. Based on the correction of traditional mode vision, the accuracy of target recognition is significantly improved, especially in complex scenarios, and more reliable target detection is achieved. The complexity and dynamic changes in the industrial environment also require the fusion system to have strong adaptability. This application can still maintain high accuracy and robustness in cases such as occlusion, defocus, and multi-target overlap.

[0048] Embodiment 1

[0049] In this embodiment, a multi-target recognition and detection method based on the fusion of UWB and machine vision is adopted to perform multi-target recognition and detection work. The method steps are as Figure 1 shown, including:

[0050] S1. Collect the three-dimensional coordinate data of the monitored object and the camera, and obtain the Euler angle parameter information of the camera;

[0051] S2. According to the three-dimensional coordinate data of the monitored object and the camera, combined with the Euler angle parameter information, calculate and obtain the pixel coordinate data of the monitored object;

[0052] S3. Use the camera to collect the monitored object, and process the collected image to obtain visual measurement data;

[0053] S4. Calculate the confidence result from the visual measurement data and the pixel coordinate data; S5. According to the confidence result, mark the target detection marking box where the confidence is greater than the preset confidence to complete multi-target recognition.

[0054] In this embodiment, first, UWB base stations are arranged in the monitoring scene to build a UWB three-dimensional space positioning coordinate system. UWB tags are carried on the monitored object and the camera to obtain the three-dimensional coordinate data of the monitored object and the camera. The camera carries a UWB six-axis gyroscope module, as Figure 4As shown in the figure, according to the range of the scene to be monitored, UWB positioning base stations are evenly distributed within the scene range, and a three-dimensional space coordinate system is established. The x-axis represents the front and back in the horizontal direction, the y-axis represents the left and right in the horizontal direction, and the z-axis represents the vertical direction, i.e., up and down. The UWB base station includes a main base station and a secondary base station, and the main base station is selected as the fixed reference point.

[0055] The monitored object and the camera carry UWB tags, and the three-dimensional coordinate data P_goal of the monitored object and the three-dimensional coordinate data P_camera of the camera are obtained. The specific expressions are as follows:

[0056] P_goal = (x, y, z) = (xg, yg, zg)

[0057] P_camera = (x', y', z') = (xc, yc, zc)

[0058] Among them, P_goal is the three-dimensional coordinate data of the monitored object; P_camera is the three-dimensional coordinate data of the camera; x, y, and z are the coordinate axis directions; xg, yg, and zg are the three coordinate axis variables of the monitored object in the three-dimensional coordinate system established by the UWB base station; xc, yc, and zc are the three coordinate axis variables of the camera in the three-dimensional coordinate system established by the UWB base station.

[0059] In this embodiment, a gyroscope module is installed on the camera, and the gyroscope module is calibrated in the UWB three-dimensional space positioning coordinate system to obtain the Euler angle parameter information of the camera in the three-dimensional space coordinate system. Among them, a gyroscope module is installed on the camera, and its initial orientation is selected as the positive Z-axis (0, 0, 1) in the three-dimensional space coordinate system established by the UWB base station, and the gyroscope module reads the rotation matrix R of the camera. Multiplying the initial camera orientation vector (0, 0, 1) by the rotation matrix R can obtain the Euler angle parameter information of the camera in the three-dimensional space coordinate system, denoted by represented.

[0060] In this embodiment, the three-dimensional coordinate data and the Euler angle parameter information of the camera are simultaneously input into the three-dimensional vision space mapping and fusion module, and the three-dimensional coordinate data of the monitored object, i.e., the target object, can be converted into pixel coordinate data in the image plane coordinate system.

[0061] First, the three-dimensional coordinate data P_goal of the monitored object is converted into camera coordinate data goal_camera. Subtracting the three-dimensional coordinate data of the monitored object from the three-dimensional coordinate data of the camera to obtain the vector P_relactive of the relative position of the camera. Among them, the vector P_relactive of the relative position of the camera is:

[0062] P_relactive = (x”, y”, z”) = (x_relactive, y_relactive, z_relactive)

[0063] x_relactive = xc - xg

[0064] y_relactive = yc - yg

[0065] z_relactive = zc - zg

[0066] Then, multiply this relative vector P_relactive by the rotation matrix R of the camera to convert the three-dimensional coordinate data of the monitored object to the camera coordinate data goal_camera. Among them, the camera coordinate data goal_camera of the monitored object is as follows:

[0067]

[0068] Among them, goal_camera is the camera coordinate data; P_relactive is the relative position vector of the camera; R is the rotation matrix of the camera; x, y, and z are the coordinate axis directions; xg, yg, and zg are the three coordinate axis variables of the monitored object in the three-dimensional coordinate system established by the UWB base station; xc, yc, and zc are the three coordinate axis variables of the camera in the three-dimensional coordinate system established by the UWB base station; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data of each direction of the relative position vector.

[0069] Then, use the parameter matrix k of the image plane where the monitored object is located collected by the camera to convert the camera coordinate data goal_camera of the monitored object to the two-dimensional image plane coordinate data goal_2Dsurface. The parameter matrix k is specifically as follows:

[0070]

[0071] Among them, hx = h / dx, hy = h / dy, the center point of the image plane is pixel_center, and pixel_center = (cx, cy).

[0072] From the Euler angle parameter information of the camera The angle θ between the two vectors can be determined by the straight-line angle formula with the vector P_relactive of the relative position of the camera; d is the distance from the camera to the monitored object; h is the distance from the three-dimensional coordinate data P_camera of the camera to the center point pixel_center of the image plane, then h = d×cos(θ), so hx and hy can be obtained.

[0073] During the normalization process: The unit of the image coordinate system is mm, which belongs to the physical unit, while the unit of the pixel coordinate system is pixel. The conversion between the two is as follows: where dx and dy represent how many mm each column and each row represent respectively, that is, 1 pixel point (pixel) = dx·mm

[0074] hx = h / dx, hy = h / dy

[0075] Among them, hx is the focal length in the x-axis direction described by pixels; hy is the focal length in the x-axis direction described by pixels, and h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane.

[0076] Project the camera coordinate data goal_camera of the object to be monitored onto the image plane, and the final calculated camera coordinate data goal_camera is converted into two-dimensional image plane coordinate data goal_2Dsurface. Among them, the two-dimensional image plane coordinate data goal_2Dsurface is:

[0077]

[0078] Among them, goal_2Dsurface is two-dimensional plane coordinate data; k is the parameter matrix of the image plane where the object to be monitored is captured by the camera; goal_camera is the camera coordinate data; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data of each direction of the relative position vector; x and y are the coordinate axes; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described by pixels; hy is the focal length in the y-axis direction described by pixels; cx and cy are the coordinates of the optical center corresponding to the principal optical axis of the camera in the pixel coordinate system.

[0079] Finally, perform normalization processing on the two-dimensional image plane coordinate data goal_2Dsurface of the object to be monitored, and convert it into the final pixel coordinate data goal_pixel in the video frame captured by the camera, and the coordinate conversion algorithm ends. The specific formula for obtaining the final pixel coordinate goal_pixel of the object to be monitored through normalization processing is:

[0080] goal_pixel = (u_final, v_final)

[0081]

[0082] Among them, goal_pixel is pixel coordinate data, and u_final and v_final are the coordinate representations of the monitored object in the pixel coordinate system; x and y are the coordinate axes; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data of each direction of the camera coordinates; R is the rotation matrix of the camera; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described in pixels; hy is the focal length in the y-axis direction described in pixels; cx and cy are the coordinates of the optical center corresponding to the principal optical axis of the camera in the pixel coordinate system.

[0083] In this embodiment, the image data collected by the camera is transmitted to the machine vision recognition module to obtain visual measurement data. Confidence is a prediction value in the machine vision module, indicating the probability that the detection box contains the target and the positioning accuracy of this box, achieving multi-class recognition, not only predicting whether there is a target, but also predicting the category of the target, involving the prediction of each category. The specific formula for confidence is:

[0084] C = P(Object) × IOU × P(Class)

[0085] Among them, C is the confidence; P(Object) is the probability that the box contains the monitored object; IOU is the intersection over union between the predicted box and the ground truth box; P(Class) is the probability that the target in the box belongs to a certain category.

[0086] The visual measurement data combined with the final pixel coordinate data of the monitored object is input into the visual confidence complementary module for processing, and the fused visual measurement data and the final pixel coordinate data of the monitored object are output.

[0087] The image plane data of the monitored object collected by the camera is transmitted to the machine vision recognition module to obtain visual measurement data, and the visual measurement data obtained by the machine vision is vision_confidence. Combining the final pixel coordinate data goal_pixel and the visual measurement data vision_confidence, complementary filtering processing is performed in the confidence complementary module. Among them, the specific steps to obtain the confidence result are: first, the pixel coordinate data is mapped to obtain the confidence complementary value of the corresponding block, and then the confidence complementary value is fused with the visual measurement data to obtain the confidence result. The specific formula for the confidence result is:

[0088] confidence_final = A · vision_confidence + (1 - A) · goal_confidence

[0089] Among them, confidence_final is the confidence result; vision_confidence is the visual measurement data; goal_confidence is the complementary confidence value, and A is the complementary filtering factor.

[0090] The preset confidence is 1 - A.

[0091] In this embodiment, for the visual measurement data, A takes 0.3.

[0092] In this method, combining the confidence result with the pixel coordinate data can significantly improve the accuracy, robustness, and real-time performance of target recognition, and at the same time support data fusion, dynamic decision-making, and system optimization.

[0093] In this embodiment, the fused visual measurement data and the pixel coordinate data of the monitored object are used to output the target detection bounding box in real time in the target marking module, and finally displayed in the display module, that is, the display effect of the camera. The fused confidence result confidence_final outputs the target detection bounding box in real time in the target marking module to ensure that only the target detection boxes with high confidence are displayed. The bounding box is superimposed on each frame of the monitoring video collected by the camera. Finally, in the display module, the video collected by the camera is synchronously displayed with the detection results.

[0094] Embodiment 2

[0095] In this embodiment, a multi-target recognition and detection system based on the fusion of UWB and machine vision is used for multi-target recognition and detection work. The system includes a data acquisition module, a data processing module, and a target recognition module; through the cooperation of each module, the detection method is realized.

[0096] Among them, the data acquisition module includes a camera, a gyroscope, a UWB positioning tag, and a UWB positioning base station. This module is used to collect the three-dimensional coordinate data of the monitored object and the camera, and obtain the Euler angle parameter information of the camera; the data processing module is used to calculate and obtain the pixel coordinate data of the monitored object according to the three-dimensional coordinate data of the monitored object and the camera, combined with the Euler angle parameter information; it is also used to process the image collected by the camera to obtain the visual measurement data; the target recognition module calculates the confidence result from the visual measurement data and the pixel coordinate data; and then according to the confidence result, marks the target detection bounding box where the confidence is greater than the preset confidence to complete multi-target recognition.

[0097] In this embodiment, the system architecture is as Figure 2As shown in the figure, the device side serves as the data acquisition module, the data processing system serves as the data processing module, and the monitoring visualization platform serves as the target recognition module. On the device side, the camera module acquires image data, the gyroscope module acquires the Euler angle values of the orientation, and the UWB positioning tag and the UWB positioning base station jointly acquire three-dimensional coordinate data. All the data is transmitted to the data processing system for coordinate conversion processing, and the machine vision recognition module in the data processing system processes it to obtain visual measurement data. The visual measurement data and the converted pixel coordinate data are transmitted to the monitoring visualization platform to calculate the confidence result, and finally, according to this confidence result, the target detection marking frame is marked in the monitoring screen.

[0098] In this embodiment, the application process of the system is as Figure 3 shown. First, UWB base stations are arranged in the monitoring scene, and a UWB three-dimensional space positioning coordinate system is established. Then, a gyroscope module is arranged on the camera, and the gyroscope module is calibrated in the UWB three-dimensional space positioning coordinate system to obtain the Euler angle of the camera's orientation in this space coordinate system. UWB tags are set on the camera and the object to be monitored to obtain the coordinates of the camera and the object to be monitored in the UWB three-dimensional space coordinate system. Then, the above coordinate information and the camera's Euler angle information of the orientation are input into the three-dimensional vision space mapping and fusion module to obtain the pixel coordinates of the target object in the image plane coordinate system. The image data of the camera is transmitted to the machine vision recognition module to obtain the visually measured data obtained by recognition, and it is transmitted to the visual confidence complementary filtering module. Then, the visual confidence complementary filtering module combines the pixel coordinate data of the target object and the recognized visual measurement data, outputs the fused visual measurement data and the pixel coordinates of the target, and finally, the visual measurement data and the pixel coordinate data of the target are input into the target marking module to output the marking frame in real time, and finally, it is displayed on the display module.

[0099] In this embodiment, first, UWB base stations are arranged in the monitoring scene, a UWB three-dimensional space positioning coordinate system is established, the object to be monitored and the camera carry UWB tags to obtain the three-dimensional coordinate data of the object to be monitored and the camera. The camera carries a UWB six-axis gyroscope module, as Figure 4 shown. According to the range of the monitoring scene required, UWB positioning base stations are evenly distributed within the scene range, a three-dimensional space coordinate system is established, the x-axis represents the front and back in the horizontal direction, the y-axis represents the left and right in the horizontal direction, and the z-axis represents the vertical direction, that is, up and down. The UWB base station includes a main base station and a secondary base station, and the main base station is selected as the fixed reference point.

[0100] The object to be monitored and the camera carry UWB tags to obtain the three-dimensional coordinate data P_goal of the object to be monitored and the three-dimensional coordinate data P_camera of the camera. Their specific expressions are:

[0101] P_goal = (x, y, z) = (xg, yg, zg)

[0102] P_camera = (x', y', z') = (xc, yc, zc)

[0103] Among them, P_goal is the three-dimensional coordinate data of the object to be monitored; P_camera is the three-dimensional coordinate data of the camera; x, y, and z are the coordinate axis directions; xg, yg, and zg are the three coordinate axis variables of the object to be monitored in the three-dimensional coordinate system established by the UWB base station; xc, yc, and zc are the three coordinate axis variables of the camera in the three-dimensional coordinate system established by the UWB base station.

[0104] In this embodiment, a gyroscope module is installed on the camera, and the gyroscope module is calibrated in the UWB three-dimensional space positioning coordinate system to obtain the Euler angle parameter information of the camera in the three-dimensional space coordinate system. Among them, a gyroscope module is installed on the camera, and its initial orientation is selected as the positive Z-axis (0, 0, 1) in the three-dimensional space coordinate system established by the UWB base station, and the gyroscope module reads the rotation matrix R of the camera. Multiplying the initial camera orientation vector (0, 0, 1) by the rotation matrix R can obtain the Euler angle parameter information of the camera in the three-dimensional space coordinate system, denoted by represented by

[0105] In this embodiment, the three-dimensional coordinate data and the Euler angle parameter information of the camera are input into the three-dimensional vision space mapping and fusion module at the same time, and the three-dimensional coordinate data of the object to be monitored, that is, the target object, can be converted into pixel coordinate data in the image plane coordinate system.

[0106] First, convert the three-dimensional coordinate data P_goal of the object to be monitored into camera coordinate data goal_camera. Subtract the three-dimensional coordinate data of the object to be monitored from the three-dimensional coordinate data of the camera to obtain the vector P_relactive of the relative position of the camera. Among them, the vector P_relactive of the relative position of the camera is:

[0107] P_relactive = (x”, y”, z”) = (x_relactive, y_relactive, z_relactive)

[0108] x_relactive = xc - xg

[0109] y_relactive = yc - yg

[0110] z_relactive = zc - zg

[0111] Then, the relative vector P_relactive is dot-multiplied by the rotation matrix R of the camera to convert the three-dimensional coordinate data of the monitored object into camera coordinate data goal_camera. Among them, the camera coordinate data goal_camera of the monitored object is as follows:

[0112]

[0113] Among them, goal_camera is the camera coordinate data; P_relactive is the relative position vector of the camera; R is the rotation matrix of the camera; x, y, and z are the coordinate axis directions; xg, yg, and zg are the three coordinate axis variables of the monitored object in the three-dimensional coordinate system established by the UWB base station; xc, yc, and zc are the three coordinate axis variables of the camera in the three-dimensional coordinate system established by the UWB base station; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data of each direction of the relative position vector.

[0114] Then, the parameter matrix k of the image plane where the monitored object is located collected by the camera is used to convert the camera coordinate data goal_camera of the monitored object into two-dimensional image plane coordinate data goal_2Dsurface. The image plane where the monitored object is located collected by the camera is as Figure 5 shown. Among them, the parameter matrix k is specifically as follows:

[0115]

[0116] Among them, hx = h / dx, hy = h / dy, the center point of the image plane is pixel_center, and pixel_center = (cx, cy).

[0117] In this embodiment, as Figure 6 shown, from the Euler angle parameter information of the camera that is, the direction vector of the camera's orientation direction and the vector P_relactive of the camera's relative position, the included angle θ between the two vectors can be determined through the straight line included angle formula. d is the distance from the three-dimensional coordinate data P_camera of the camera to the center point pixel_center of the image plane, then h = d×cos(θ), so hx and hy can be obtained.

[0118] The camera coordinate data goal_camera of the monitored object is projected onto the image plane to obtain the final calculated camera coordinate data goal_camera converted into two-dimensional image plane coordinate data goal_2Dsurface. Among them, the two-dimensional image plane coordinate data goal_2Dsurface is as follows:

[0119]

[0120] Among them, goal_2Dsurface is the two-dimensional plane coordinate data; k is the parameter matrix of the image plane where the monitored object is located collected by the camera; goal_camera is the camera coordinate data; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data in each direction of the camera coordinates; x and y are the coordinate axes; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described in pixels; hy is the focal length in the y-axis direction described in pixels; cx and cy are the coordinates of the optical center corresponding to the principal optical axis of the camera in the pixel coordinate system.

[0121] Finally, the two-dimensional image plane coordinate data goal_2Dsurface of the monitored object is normalized and converted into the final pixel coordinate data goal_pixel in the video frame collected by the camera, and the coordinate conversion algorithm ends. The specific formula for obtaining the final pixel coordinate goal_pixel of the monitored object through normalization is:

[0122] goal_pixel = (u_final, v_final)

[0123]

[0124] Among them, goal_pixel is the pixel coordinate data, u_final and v_final are the coordinate representations of the monitored object in the pixel coordinate system; x and y are the coordinate axes; x_goal_camera, y_goal_camera, and z_goal_camera are the coordinate data in each direction of the camera coordinates; R is the rotation matrix of the camera; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described in pixels; hy is the focal length in the y-axis direction described in pixels; cx and cy are the coordinates of the optical center corresponding to the principal optical axis of the camera in the pixel coordinate system.

[0125] In this embodiment, the image data collected by the camera is transmitted to the machine vision recognition module to obtain visual measurement data. Confidence is a prediction value in the machine vision module, indicating the probability that the detection box contains the target and the positioning accuracy of this box. To achieve multi-class recognition, it not only predicts whether there is a target but also predicts the category of the target, involving the prediction of each category. The specific formula for confidence is:

[0126] C = P(Object) × IOU × P(Class)

[0127] Among them, C is the confidence; P(Object) is the probability that the object to be monitored is included in the box; IOU is the intersection over union between the predicted box and the ground truth box; P(Class) is the probability that the object in the box belongs to a certain category.

[0128] The visual measurement data and the final pixel coordinate data of the object to be monitored are input into the visual confidence complementary module for processing, and the fused visual measurement data and the final pixel coordinate data of the object to be monitored are output.

[0129] The image plane data of the object to be monitored collected by the camera is transmitted to the machine vision recognition module to obtain visual measurement data, and the visual measurement data obtained by the machine vision is vision_confidence. Combining the final pixel coordinate data goal_pixel and the visual measurement data vision_confidence, complementary filtering processing is performed in the confidence complementary module. The specific steps to obtain the confidence result are as follows: First, the pixel coordinate data is mapped to obtain the confidence complementary value of the corresponding block, and then the confidence complementary value is fused with the visual measurement data to obtain the confidence result. Among them, the specific formula for the confidence result is:

[0130] confidence_final = A · vision_confidence+(1 - A) · goal_confidence

[0131] Among them, confidence_final is the confidence result; vision_confidence is the visual measurement data; goal_confidence is the confidence complementary value, and A is the complementary filtering factor.

[0132] The preset confidence is 1 - A.

[0133] In this embodiment, A takes 0.3.

[0134] In this embodiment of the visual measurement data, the fused visual measurement data and the pixel coordinate data of the object to be monitored are used to output the target detection marking box in real time in the target marking module, and finally, it is displayed in the display module, that is, the display effect of the camera. The fused confidence result confidence_final outputs the target detection marking box in real time in the target marking module to ensure that only the target detection boxes with high confidence are displayed. The marking box is superimposed on each frame of the monitoring video collected by the camera. Finally, in the display module, the video collected by the camera and the detection results are displayed synchronously.

[0135] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A multi-target recognition and detection method based on the fusion of UWB and machine vision, characterized in that: In this method, UWB positioning base stations are evenly arranged within the preset monitored scene range, and cameras are set to identify the monitored objects. Both the monitored objects and the cameras carry UWB tags. The method steps include: S1, collecting the three-dimensional coordinate data of the monitored object and the camera, and obtaining the Euler angle parameter information of the camera; S2. Calculate the pixel coordinate data of the monitored object based on the three-dimensional coordinate data of the monitored object and the camera and the Euler angle parameter information; S3, using a camera to capture images of the monitored object, and processing the captured images to obtain visual measurement data; S4, calculating the confidence result based on the visual measurement data and the pixel coordinate data; S5. According to the confidence result, mark the target detection mark box where the confidence is greater than the preset confidence to complete multi-target recognition.

2. According to claim 1, a multi-target recognition and detection method based on the fusion of UWB and machine vision is characterized in that: The coordinate system adopted by the three-dimensional coordinate data in S1 is a three-dimensional space coordinate system established with the main base station in the UWB positioning base station as a fixed reference point. The specific coordinates of the three-dimensional coordinate data of the monitored object and the camera collected are: P_goal=(x,y,z)=(xg,yg,zg) P_camera=(x',y',z')=(xc,yc,zc) Among them, P_goal is the three-dimensional coordinate data of the monitored object; P_camera is the three-dimensional coordinate data of the camera; xg, yg and zg are the three coordinate axis variables in the three-dimensional coordinate system established by the monitored object in the UWB base station; xc, yc and zc are the three coordinate axis variables in the three-dimensional coordinate system established by the camera in the UWB base station.

3. The multi-target recognition and detection method based on the fusion of UWB and machine vision according to claim 2 is characterized in that: The specific process of obtaining the Euler angle parameter information of the camera in S1 is: setting the initial orientation of the camera to the positive Z axis (0,0,1) in the established three-dimensional space coordinate system, collecting the rotation matrix of the camera, and multiplying the initial camera orientation vector (0,0,1) by the rotation matrix to obtain the Euler angle parameter information of the camera.

4. The multi-target recognition and detection method based on the fusion of UWB and machine vision according to claim 1 is characterized in that: The steps of obtaining the pixel coordinate data of the monitored object in S2 are: first, using the three-dimensional coordinate data of the camera and the monitored object to obtain the relative position vector, then using the relative position vector to convert the three-dimensional coordinate data of the monitored object into camera coordinate data, then using the Euler angle parameter information of the camera to convert the camera coordinate data of the monitored object into two-dimensional plane coordinate data, and finally normalizing the two-dimensional plane coordinate data of the monitored object to obtain the pixel coordinate data of the monitored object.

5. The multi-target recognition and detection method based on the fusion of UWB and machine vision according to claim 4 is characterized in that: The specific formula for converting the three-dimensional coordinate data of the monitored object into the camera coordinate data is: Among them, goal_camera is the camera coordinate data; P_relactive is the camera relative position vector; R is the rotation matrix of the camera; x, y and z are the coordinate axis directions; xg, yg and zg are the three coordinate axis variables in the three-dimensional coordinate system established by the monitored object in the UWB base station; xc, yc and zc are the three coordinate axis variables in the three-dimensional coordinate system established by the camera in the UWB base station; x_goal_camera, y_goal_camera and z_goal_camera are the coordinate data of the relative position vector in each direction.

6. The multi-target recognition and detection method based on the fusion of UWB and machine vision according to claim 4 is characterized in that: The specific formula for converting the camera coordinate data of the monitored object into two-dimensional plane coordinate data is: Among them, goal_2Dsurface is the two-dimensional plane coordinate data; k is the parameter matrix of the image plane where the monitored object is located captured by the camera; goal_camera is the camera coordinate data; x_goal_camera, y_goal_camera and z_goal_camera are the coordinate data of the relative position vector in each direction; x and y are the coordinate axis directions; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described by pixels; hy is the focal length in the y-axis direction described by pixels; cx and cy are the coordinates of the optical center corresponding to the main optical axis of the camera in the pixel coordinate system.

7. The multi-target recognition and detection method based on the fusion of UWB and machine vision according to claim 4 is characterized in that: The specific formula for normalizing the two-dimensional plane coordinate data of the monitored object is: goal_pixel=(u_final,v_final) Among them, goal_pixel is the pixel coordinate data, u_final and v_final are the coordinate representations of the monitored object in the pixel coordinate system; x and y are the coordinate axis directions; x_goal_camera, y_goal_camera and z_goal_camera are the coordinate data of the relative position vector in each direction; R is the rotation matrix of the camera; h is the distance from the three-dimensional coordinate data of the camera to the center point of the image plane; hx is the focal length in the x-axis direction described by pixels; hy is the focal length in the y-axis direction described by pixels; cx and cy are the coordinates of the optical center corresponding to the main optical axis of the camera in the pixel coordinate system. goal_pixel obtains the complementary value of the confidence of the corresponding block, goal_confidence, through mapping.

8. The multi-target recognition and detection method based on the fusion of UWB and machine vision according to claim 1 is characterized in that: The specific steps of S4 are: firstly, the pixel coordinate data is mapped to obtain the confidence complementary value of the corresponding block, and then the confidence complementary value is integrated with the visual measurement data to obtain the confidence result, wherein the specific formula of the confidence result is: confidence_final=A·vision_confidence+(1-A)·goal_confidence Among them, confidence_final is the confidence result; vision_confidence is the visual measurement data; goal_confidence is the complementary value of confidence, and A is the complementary filter factor.

9. The multi-target recognition and detection method based on the fusion of UWB and machine vision according to claim 8 is characterized in that: The preset confidence in S4 is 1-A.

10. A multi-target recognition and detection system based on the fusion of UWB and machine vision, characterized in that: The system works by using a multi-target recognition and detection method based on the fusion of UWB and machine vision as described in any one of claims 1 to 9, and the system includes a data acquisition module, a data processing module, and a target recognition module; The data acquisition module includes a camera, a gyroscope, a UWB positioning tag and a UWB positioning base station. The module is used to collect the three-dimensional coordinate data of the monitored object and the camera, and obtain the Euler angle parameter information of the camera; The data processing module is used to calculate the pixel coordinate data of the monitored object based on the three-dimensional coordinate data of the monitored object and the camera in combination with the Euler angle parameter information; It is also used to process the images acquired by the camera to obtain visual measurement data; The target recognition module calculates the confidence result from the visual measurement data and the pixel coordinate data; Then, based on the confidence results, the target detection mark boxes where the confidence is greater than the preset confidence are marked to complete multi-target recognition.

Images