Dynamic measurement helmet based on vision and posture fusion

Abstract

The present invention relates to a dynamic measurement helmet based on vision and posture fusion, characterized in that it includes an image feature extraction module for binocular vision imaging, an image feature matching module for binocular vision imaging, and a size calculation module based on key point image coordinates. The present invention solves the key problem that when the camera is not fixed, the external parameters cannot be determined, resulting in the inability to map image pixels from the image coordinate system to the real-world coordinate system. The present invention realizes a safety helmet with an integrated non-contact measurement function, which can achieve a certain degree of precision in measuring objects in the target scene. The present invention can realize a safety helmet with an integrated non-contact measurement function, which can achieve a certain degree of precision in measuring objects in the target scene; the present invention can achieve accurate extraction and accurate matching of image features under conditions such as high noise interference in industrial environments.

Description

Technical Field

[0001] The invention relates to a dynamic measurement helmet based on vision and posture fusion, and belongs to the field of information processing. Background Art

[0002] With the development of computer technology, visual measurement has become a key area of measurement development in recent years. Compared to non-contact measurement methods such as laser measurement, visual measurement is less expensive, more versatile, and highly applicable to different objects in different scenarios. Monocular measurement systems based on a single camera solve the problem of measuring the two-dimensional dimensions of simple, regular objects on a plane. However, due to technical limitations, they must be fixed in position and calibrated after each installation, making the process cumbersome and difficult to achieve out-of-the-box usability. Binocular vision measurement systems, on the other hand, use the parallax between two cameras to determine the distance between the object being measured and the measuring camera, thereby providing support for calculating the object's three-dimensional dimensions.

[0003] Research on binocular stereo vision technology began early abroad, and after decades of development and research, it has now achieved numerous mature practical applications. For example, Dhaval K. Patel et al. used Matlab to design algorithms and developed a stereo vision measurement system. This system, using industrial camera calibration and triangulation principles, achieves high-precision distance measurement. Curiosity, the world's first nuclear-powered Mars rover, is a new-generation rover built by NASA. Equipped with visual sensors, it uses a binocular stereo vision system for image acquisition, constructing a three-dimensional image of the Martian surface environment to facilitate path planning.

[0004] China started later than other countries, with the application of vision technology beginning in the 1990s, and machine vision was still an emerging industry. Tsinghua University's National Laboratory for Information Science and Technology conducted research on 3D modeling methods based on stereo vision. Leveraging linear structured light and stereo vision principles, they proposed a new calibration algorithm and efficient surface reconstruction algorithm for binocular industrial cameras based on multiple free-form planes. They also constructed an automatic response modeling prototype system for virtual reality applications. Quan Yanming and others built a binocular measurement system for CNC machine tools in workshops. By moving the camera to capture images of different parts of the workpiece, they achieved geometric dimensional measurement of simple workpieces.

[0005] Despite years of development and research, binocular stereo vision has achieved significant progress and research results. However, since our research is still in its early stages, both in theory and in practical production applications, some challenges remain. For example, despite current technological advances and advanced hardware, the human visual system, consisting of both eyes, still struggles to process and calculate images captured by the outside world to ultimately identify the target object. Summary of the Invention

[0006] The purpose of the present invention is to implement an out-of-the-box non-contact measurement method based on a safety helmet with integrated cameras and other multi-sensors, mainly for scenarios such as large-scale outdoor equipment or batch sampling of assembly line products, which are difficult to measure using contact methods and inconvenient to spend a lot of time installing and debugging fixed non-contact measurement systems.

[0007] In order to achieve the above objectives, the technical solution of the present invention is to provide a dynamic measurement helmet based on vision and posture fusion, characterized in that an image feature extraction module integrating binocular vision imaging, an image feature matching module integrating binocular vision imaging, and a size calculation module based on key point image coordinates are installed on the helmet, wherein:

[0008] The image feature extraction module of binocular vision imaging is used to extract feature points from the binocular vision image to obtain a feature point set;

[0009] The image feature matching module of binocular vision imaging is used to establish a binocular system model based on the correspondence between feature point sets, thereby calculating depth or camera pose;

[0010] After obtaining the key point positions of the target on the image using the image feature extraction module and the image feature matching module, the binocular camera is calibrated by the size calculation module based on the key point image coordinates to obtain the camera's intrinsic parameters and initial extrinsic parameters, and the camera's rotation matrix relative to the initial state is calculated based on the posture sensor parameters; the size calculation module then uses matrix operations to obtain the actual rotation matrix of the camera under this posture combined with the depth information of the key points; finally, the size calculation module obtains the coordinates of all key points in the real three-dimensional space according to the coordinate transformation formula, thereby measuring the target size.

[0011] Preferably, the image feature extraction module includes a deep neural network, which includes a basic network layer and a post-processing layer, wherein:

[0012] The basic network layer is used to process the binocular vision image to obtain the feature map;

[0013] The post-processing layer further processes the feature map to obtain a set of feature points.

[0014] Preferably, the post-processing layer includes an interpolation scale-up unit, a convolution dimension reduction and normalization unit, and a convolution pooling unit, wherein:

[0015] The interpolation scale magnification unit magnifies the feature map at different scales through bilinear interpolation;

[0016] The convolution dimension reduction and normalization unit further processes the feature map output by the interpolation scale amplification unit to obtain an output feature map. The further processing includes the following steps:

[0017] Perform two-dimensional convolution on each scaled feature map and reduce the dimensionality to 16 channels. After completing the multi-scale transformation and convolution layer perception, use bilinear interpolation on the feature map again to change its size. Finally, concatenate the multi-scale feature maps and fuse them into a whole feature map to obtain the output feature map.

[0018] The convolution pooling unit is used to perform convolution pooling on the output feature map to obtain the output of the final feature point set.

[0019] Preferably, the convolution pooling unit uses a 3D convolution layer and a 3D maximum pooling layer to perceive the output feature map, and finally outputs a feature map with a channel number of 1, and then extracts the feature point set through the Top_K method.

[0020] Preferably, the image feature matching module is implemented based on a random sampling consensus algorithm, which calculates mathematical model parameters of the data based on a set of sample data sets containing abnormal data to obtain valid sample data, including the following steps:

[0021] Step 1: Randomly select a RANSAC sample from the sample set, that is, 4 matching point pairs;

[0022] Step 2: Calculate the transformation matrix M based on these four matching point pairs;

[0023] Step 3: Calculate the consensus set that satisfies the current transformation matrix M based on the sample set, the transformation matrix M, and the error metric function, and return the number of elements in the consensus set consensus.

[0024] Step 4: Determine whether it is the optimal consensus set based on the number of elements in the current consensus set. If so, update the current optimal consensus set.

[0025] Step 5: Update the current error probability p. If p is greater than the minimum allowed error probability, repeat steps 1 to 4 and continue iterating until the current error probability p is less than the minimum error probability.

[0026] Preferably, the size calculation module calibrates the binocular camera using the following steps:

[0027] The Zhang Zhengyou calibration method uses a checkerboard grid to calibrate each camera's intrinsic parameters individually. A checkerboard grid is printed and fixed flat on a surface as a calibration object. Twenty photos of the calibration object are taken from different orientations by adjusting the camera's position. Checkerboard corner points are extracted from each photo to estimate the intrinsic parameters for an ideal, undistorted condition and the initial extrinsic parameters for a horizontal camera placement on the helmet. The least squares method is then used to estimate the distortion parameters for a realistic situation, considering only radial distortion. Finally, the maximum likelihood method is used to optimize the estimates and improve accuracy.

[0028] Preferably, the size calculation module calculates the rotation matrix of the camera relative to the initial state according to the posture sensor parameters by the following method:

[0029] Initialize the posture sensor fixed on the helmet when the helmet is placed horizontally. Then use the sensor to obtain the rotation angles β, φ, and θ around the X, Y, and Z axes relative to the current initial position. The corresponding rotation transformation matrix is: The calculated rotation matrix of the overall binocular camera relative to the initial state is R t =Rot(X,β)Rot(X,φ)Rot(X,θ).

[0030] Preferably, the size calculation module obtains depth information by the following method: based on the coordinates of the matching key points obtained from the images of the target object at different angles in the same scene obtained by the binocular camera, the depth information is obtained by calculating the parallax between the two, wherein the X axes of the binocular cameras coincide with each other, the optical axes are parallel to each other, the distance between the optical centers of the left and right cameras in the binocular camera is T, and the focal lengths f of the two cameras are l =f r =f, then the imaging points of point P in three-dimensional space on the left and right cameras are p l and p r , the parallax of point P is d=|x l -x r |, the depth information of point P can be obtained from similar triangles

[0031] Preferably, the coordinate conversion formula used by the size calculation module is:

[0032]

[0033] Where u and v are the coordinates of the key target point in the pixel coordinate system; dx and dy are the physical dimensions of a pixel in the u-axis and v-axis directions; f is the focal length of the binocular camera; R is the real-time rotation transformation matrix calculated based on the pose sensor parameters and the initial rotation matrix calibrated when the helmet is placed horizontally; T is the translation transformation matrix during calibration; This is the internal parameter of the calibrated double-sided camera; is the calculated double-sided camera extrinsic parameter when the helmet is in the current position; H = NW; (X w ,Y w ,Z W ) is the coordinate of the key target point in the real three-dimensional space; u o 、v o is the pixel coordinate.

[0034] Preferably, the size calculation module obtains the real-time rotation transformation matrix R by the following method:

[0035] The size calculation module based on the key point image coordinates obtains the rotation matrix R of the helmet relative to the horizontal position when the posture sensor takes the picture t , combined with the rotation matrix R0 in the initial external parameters of the binocular camera, use the matrix operation R=R0R t Get the camera extrinsic rotation matrix R under this pose.

[0036] This invention solves the key problem of being unable to map image pixels from the image coordinate system to the real-world coordinate system due to the inability to determine extrinsic parameters when the camera is not fixed. This invention also implements a safety helmet with integrated non-contact measurement capabilities, enabling accurate measurement of objects in target scenes.

[0037] Compared with the prior art, the present invention has the following technical effects:

[0038] (1) The present invention can realize a safety helmet with an integrated non-contact measurement function, which can measure objects in the target scene with a certain degree of accuracy;

[0039] (2) The present invention can accurately extract and match image features under conditions of high noise interference in industrial environments;

[0040] (3) The present invention can realize real-time measurement of the size of the target object based on the key points of feature matching by using a non-fixed position measurement system composed of a binocular camera and a posture sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the feature detection and stereo matching method of binocular vision imaging of the present invention;

[0042] Figure 2 Schematic diagram of the method for obtaining depth information of target key points. DETAILED DESCRIPTION

[0043] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0044] This embodiment discloses a dynamic measurement helmet based on vision and posture fusion, including a binocular vision imaging image feature extraction module, a binocular vision imaging image feature matching module, and a size calculation module based on key point image coordinates.

[0045] The image feature extraction module of binocular vision imaging is used to extract relatively special and easily distinguishable feature points, such as corner points, from the binocular vision image. The image feature extraction module is mainly composed of a deep neural network. The deep neural network used in this embodiment is divided into a basic network layer and a post-processing layer. The basic network layer uses the basic network structure of ResNet18. The basic network layer can process the binocular vision image to obtain a feature map. The post-processing layer further processes the feature map to obtain a feature point set. The post-processing layer further includes an interpolation scale magnification unit, a convolution dimensionality reduction normalization unit, and a convolution pooling unit.

[0046] The interpolation scale magnification unit mainly realizes the magnification of different scales of the feature map through bilinear interpolation. In this embodiment, four scale ratios evenly distributed between 1 and 10 times are used to process the original feature map, and the processing method is bilinear interpolation.

[0047] The convolutional dimension reduction and normalization unit further processes the feature map output by the interpolation and scaling unit to obtain an output feature map. In this embodiment, the convolutional dimension reduction and normalization unit performs two-dimensional convolution on each scaled feature map and reduces the dimension to 16 channels. Finally, after completing the multi-scale transformation and convolutional layer perception, bilinear interpolation is used again on the feature map to change its size. To facilitate subsequent operations, this embodiment sets it to the original feature map size, and finally connects the multi-scale feature maps and fuses them into a single feature map.

[0048] The convolutional pooling unit is used to perform convolutional pooling on the output feature map to obtain the final output feature point set. To comprehensively understand the information of all features, this embodiment uses a 3D convolutional layer and a 3D max pooling layer to perceive the input feature map, ultimately outputting a feature map with one channel, and extracting the feature point set using the Top_K method.

[0049] The image feature matching module of binocular vision imaging is used to establish a binocular system model based on the correspondence between feature points or feature areas, thereby calculating the depth or camera posture. In this embodiment, the image feature matching module is mainly implemented by the random sampling consensus algorithm (RANSAC). The random sampling consensus algorithm calculates the mathematical model parameters of the data based on a set of sample data sets containing abnormal data to obtain valid sample data. The main process of the random sampling consensus algorithm is as follows:

[0050] Step 1: Randomly select a RANSAC sample from the sample set, that is, 4 matching point pairs;

[0051] Step 2: Calculate the transformation matrix M based on these four matching point pairs;

[0052] Step 3: Calculate the consensus set that satisfies the current transformation matrix M based on the sample set, the transformation matrix M, and the error metric function, and return the number of elements in the consensus set consensus.

[0053] Step 4: Determine whether it is the optimal consensus set based on the number of elements in the current consensus set. If so, update the current optimal consensus set.

[0054] Step 5: Update the current error probability p. If p is greater than the minimum allowed error probability, repeat steps 1 to 4 and continue iterating until the current error probability p is less than the minimum error probability.

[0055] After obtaining the key point positions of the target on the image using the image feature extraction module and the image feature matching module, the binocular camera is calibrated by the size calculation module based on the key point image coordinates to obtain the camera intrinsic parameters and initial extrinsic parameters. The camera rotation matrix relative to the initial state is calculated based on the pose sensor parameters. Matrix operations are then used to obtain the actual rotation matrix of the camera in this pose, combined with the depth information of the key points. Finally, the coordinates of all key points in real three-dimensional space can be obtained according to the coordinate transformation formula to measure the target size.

[0056] The size calculation module, based on keypoint image coordinates, calibrates the binocular camera using the following method: Each camera's intrinsic parameters are individually calibrated using a checkerboard grid, using Zhang Zhengyou's calibration method. A checkerboard grid is printed and fixed flat on a surface as a calibration object. Twenty photos of the calibration object are taken from different orientations by adjusting the camera's position. Checkerboard corners are extracted from each photo to estimate the intrinsic parameters for an ideal, undistorted scenario and the initial extrinsic parameters for a horizontal camera placement on the helmet. The least squares method is then used to estimate the distortion parameters for a realistic scenario, considering only radial distortion. Finally, the maximum likelihood method is used to optimize the estimates and improve accuracy.

[0057] The size calculation module based on the key point image coordinates calculates the rotation matrix of the camera relative to the initial state according to the posture sensor parameters by the following method: the posture sensor fixed on the helmet is initialized when the helmet is placed horizontally, and then the rotation angles β, φ, and θ around the X, Y, and Z axes relative to the current initial position are obtained through the sensor. The corresponding rotation transformation matrix is The calculated rotation matrix of the overall binocular camera relative to the initial state is R t =Rot(X,β)Rot(X,φ)Rot(X,θ).

[0058] The size calculation module based on the key point image coordinates obtains the depth information by the following method: Based on the images of the target object at different angles in the same scene obtained by the binocular camera, the coordinates of the matching key points are obtained, and the depth information is obtained by calculating the parallax between the two. The calculation method is as follows: Figure 1 As shown in the figure, the X axes of the binocular cameras coincide with each other, the optical axes are parallel to each other, the distance between the optical centers of the left and right cameras in the binocular camera is T, and the focal lengths of the two cameras are f l =f r =f. The imaging points of a point P in three-dimensional space on the left and right cameras are p l and p r , then the parallax of point P is d=|x l -x r |, the depth information of the point can be obtained from similar triangles

[0059] The coordinate conversion formula used by the size calculation module based on the key point image coordinates is:

[0060]

[0061] Where u and v are the coordinates of the key target point in the pixel coordinate system; dx and dy are the physical dimensions of a pixel in the u-axis and v-axis directions; f is the focal length of the binocular camera; R is the real-time rotation transformation matrix calculated based on the pose sensor parameters and the initial rotation matrix calibrated when the helmet is placed horizontally; T is the translation transformation matrix during calibration; This is the internal parameter of the calibrated double-sided camera; is the calculated double-sided camera extrinsic parameter when the helmet is in the current position; H = NW; (X w ,Y w ,Z W ) is the coordinate of the key target point in the real three-dimensional space; u o 、v o is the pixel coordinate.

[0062] The size calculation module based on the key point image coordinates obtains the rotation matrix R of the helmet relative to the horizontal position when the posture sensor takes the picture t , combined with the rotation matrix R0 in the initial external parameters of the binocular camera, use the matrix operation R=R0R t The camera external parameter rotation matrix R under this posture is obtained, and finally the coordinates of the key target points in the real world coordinates are calculated through the coordinate transformation formula to measure the target size.

Claims

1. A dynamic measurement helmet based on vision and posture fusion, characterized in that: The helmet is equipped with an image feature extraction module integrating binocular vision imaging, an image feature matching module integrating binocular vision imaging, and a size calculation module based on key point image coordinates, wherein: The image feature extraction module of binocular vision imaging is used to extract feature points from the binocular vision image to obtain a feature point set; The image feature matching module of binocular vision imaging is used to establish a binocular system model based on the correspondence between feature point sets, thereby calculating depth or camera pose; After obtaining the key point positions of the target on the image using the image feature extraction module and the image feature matching module, the binocular camera is calibrated by the size calculation module based on the key point image coordinates to obtain the camera's intrinsic parameters and initial extrinsic parameters. The camera's rotation matrix relative to the initial state is calculated based on the pose sensor parameters. The size calculation module then uses matrix operations to obtain the actual rotation matrix of the camera in this pose, combined with the depth information of the key points. Finally, the size calculation module obtains the coordinates of all key points in real three-dimensional space based on the coordinate transformation formula, thereby measuring the target size. The image feature matching module is implemented based on a random sampling consensus algorithm. The random sampling consensus algorithm calculates the mathematical model parameters of the data based on a set of sample data sets containing abnormal data to obtain valid sample data, including the following steps: Step 1: Randomly select a RANSAC sample from the sample set, that is, 4 matching point pairs; Step 2: Calculate the transformation matrix M based on these four matching point pairs; Step 3: Calculate the consensus set that satisfies the current transformation matrix M based on the sample set, the transformation matrix M, and the error metric function, and return the number of elements in the consensus set consensus. Step 4: Determine whether it is the optimal consensus set based on the number of elements in the current consensus set. If so, update the current optimal consensus set. Step 5: Update the current error probability p. If p is greater than the minimum error probability allowed, repeat steps 1 to 4 until the current error probability p is less than the minimum error probability. The size calculation module calculates the rotation matrix of the camera relative to the initial state based on the pose sensor parameters using the following method: Initialize the posture sensor fixed on the helmet when the helmet is placed horizontally. Then use the sensor to obtain the rotation angles β, φ, and θ around the X, Y, and Z axes relative to the current initial position. The corresponding rotation transformation matrix is: The calculated rotation matrix of the overall binocular camera relative to the initial state is R t =Rot(X,β)Rot(X,φ)Rot(X,θ).

2. A dynamic measurement helmet based on vision and posture fusion as claimed in claim 1, characterized in that: The image feature extraction module includes a deep neural network, which includes a basic network layer and a post-processing layer, wherein: The basic network layer is used to process the binocular vision image to obtain the feature map; The post-processing layer further processes the feature map to obtain a set of feature points.

3. A dynamic measurement helmet based on vision and posture fusion as claimed in claim 2, characterized in that: The post-processing layer includes an interpolation scale-up unit, a convolution dimension reduction and normalization unit, and a convolution pooling unit, wherein: The interpolation scale magnification unit magnifies the feature map at different scales through bilinear interpolation; The convolution dimension reduction and normalization unit further processes the feature map output by the interpolation scale amplification unit to obtain an output feature map. The further processing includes the following steps: Perform two-dimensional convolution on each scaled feature map and reduce the dimensionality to 16 channels. After completing the multi-scale transformation and convolution layer perception, use bilinear interpolation on the feature map again to change its size. Finally, concatenate the multi-scale feature maps and fuse them into a whole feature map to obtain the output feature map. The convolution pooling unit is used to perform convolution pooling on the output feature map to obtain the output of the final feature point set.

4. A dynamic measurement helmet based on vision and posture fusion as claimed in claim 3, characterized in that: The convolution pooling unit uses a 3D convolution layer and a 3D maximum pooling layer to perceive the output feature map, and finally outputs a feature map with a channel number of 1, and then extracts the feature point set through the Top_K method.

5. The dynamic measurement helmet based on vision and posture fusion according to claim 1, characterized in that: The size calculation module calibrates the binocular camera using the following steps: Zhang Zhengyou's calibration method is used to calibrate the intrinsic parameters of each camera using a checkerboard grid. A checkerboard is printed and fixed flat on a plane as a calibration object. By adjusting the camera position, 20 photos of the calibration object are taken in different directions. The checkerboard corner points are extracted from each photo to estimate the intrinsic parameters under the ideal distortion-free condition and the initial extrinsic parameters when the camera is placed horizontally on the helmet. The least squares method is then used to estimate the distortion parameters under the actual condition of only considering radial distortion. Finally, the maximum likelihood method is used to optimize the estimation and improve accuracy.

6. The dynamic measurement helmet based on vision and posture fusion according to claim 1, characterized in that: The size calculation module obtains depth information by the following method: based on the coordinates of the matching key points obtained from the images of the target object at different angles in the same scene obtained by the binocular camera, the depth information is obtained by calculating the parallax between the two, wherein the X axes of the binocular cameras coincide with each other, the optical axes are parallel to each other, the distance between the optical centers of the left and right cameras in the binocular camera is T, and the focal lengths of the two cameras are f l =f r =f, then the imaging points of point P in three-dimensional space on the left and right cameras are p l and p r , the parallax of point P is d=|x l -x r |, the depth information of point P can be obtained from similar triangles 7. The dynamic measurement helmet based on vision and posture fusion according to claim 1, characterized in that: The coordinate conversion formula used by the size calculation module is: Where u and v are the coordinates of the key target point in the pixel coordinate system; dx and dy are the physical dimensions of a pixel in the u-axis and v-axis directions; f is the focal length of the binocular camera; R is the real-time rotation transformation matrix calculated based on the pose sensor parameters and the initial rotation matrix calibrated when the helmet is placed horizontally; T is the translation transformation matrix during calibration; This is the internal parameter of the calibrated double-sided camera; is the calculated double-sided camera extrinsic parameter when the helmet is in the current position; H = NW; (X w ,Y w ,Z W ) is the coordinate of the key target point in the real three-dimensional space; u o 、v o is the pixel coordinate.

8. A dynamic measurement helmet based on vision and posture fusion as claimed in claim 7, characterized in that: The size calculation module obtains the real-time rotation transformation matrix R by the following method: The size calculation module based on the key point image coordinates obtains the rotation matrix R of the helmet relative to the horizontal position when the posture sensor takes the picture t , combined with the rotation matrix R0 in the initial external parameters of the binocular camera, use the matrix operation R=R0R t Get the camera extrinsic rotation matrix R under this pose.

Images