A guide for the blind based on deep image segmentation and human model orthographic projection algorithm

Abstract

The application discloses a kind of based on depth image segmentation and human body model orthographic projection algorithm's guide blind method, this method includes the following steps: the body related size data and the image data of the front road obstacle condition of user are obtained by depth camera;Set initial threshold distance and minimum safety distance, the object in initial threshold distance is projected to plumb plane, and the plane visual barrier chart is constructed;Judge whether there is passable area in visual barrier chart;If there is passable area, then guide user to walk to the area, if there is no passable area, then reduce current distance threshold and repeat the above steps, if current distance threshold is reduced to below minimum safety distance threshold, then judge no passable area, issue turning instruction.The application is based on depth image segmentation, and the depth image obtained by depth camera is divided into passable area according to the position of obstacle and provides relevant guide blind information, to further improve the accuracy of guide blind obstacle avoidance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image target detection, and particularly relates to a guide blind method based on a depth image segmentation and human body model orthographic projection algorithm. BACKGROUND

[0002] With the rapid development of public facilities in China, visually impaired people still face many difficulties in travel, including unreasonable planning of blind paths, unclear road guidance and other problems. Although our society has gradually made progress in the popularization of barrier-free facilities, these facilities still cannot fully meet the actual needs of visually impaired people, resulting in great travel challenges for them in daily life.

[0003] In today's society, the main travel aids for visually impaired people mainly include guide canes and guide dogs. However, both of them have their own limitations, making them unable to completely solve the navigation needs of visually impaired people in different environments. Guide canes perform well in familiar environments, but due to the lack of advanced detection functions, they are not up to the task when faced with complex road conditions. Guide dogs have relatively good assistance, but due to the high training cost and insufficient quantity, they are far from meeting the needs of more than 23 million visually impaired people in China.

[0004] Under this background, existing guide products are increasingly difficult to meet market demand, and there is an urgent need for more advanced guide devices to fully assist the travel of visually impaired people. The current guide method mainly uses target detection algorithm and SLAM (Simultaneous Localization and Mapping) algorithm. However, both methods have a series of defects, which makes them unable to provide satisfactory navigation experience in all situations. The target detection algorithm performs well in specific scenarios, but its adaptability in unfamiliar environments is poor, it is easily affected by environmental complexity, and it requires high computing resources, making it difficult to achieve efficient deployment on resource-constrained devices such as embedded systems; the SLAM algorithm solves the positioning and map construction problem in navigation to some extent, but it still has certain limitations in object recognition, and it is difficult to accurately perceive and recognize obstacles on the road, limiting its reliability in actual travel. SUMMARY

[0005] The purpose of the present application is to solve the above-mentioned defects in the prior art, and to provide a guide blind method based on depth image segmentation and human body model orthographic projection algorithm, which can improve the accuracy of identification by identifying the road conditions and obstacles through a depth camera.

[0006] The purpose of the present application can be achieved by adopting the following technical solutions:

[0007] A blind guiding method based on a depth image segmentation and a human body model orthographic projection algorithm, the blind guiding method comprising the following steps:

[0008] S1, calibrating a depth camera of a blind guiding device using a small cube with a side length of a to obtain a scaling ratio f(d), d being a depth value of a depth image obtained by the depth camera, a user standing facing the depth camera to obtain a minimum value d0 of the depth value in a human body pixel point, a horizontal pixel width n x , and a vertical pixel height n y , substituting d0, n x , n y into the scaling ratio f(d) to obtain a height h1 of the user and a shoulder width w1;

[0009] S2, setting an initial threshold distance D1 and a minimum safety distance c1, setting a current threshold distance as D, constructing a human body orthographic projection matrix A m×n according to the scaling ratio f(d) and the height h1 and the shoulder width w1 of the user, m being a number of pixel points of a human body model in a vertical direction of the matrix, and n being a number of pixel points of the human body model in a horizontal direction of the matrix;

[0010] S3, setting a standard coordinate system O uvw , wherein u is a horizontal direction, v is a vertical direction, w is a plumb direction, and a ground is a u-w plane, performing a rigid transformation on an internal parameter view matrix V in of the depth camera through a coordinate system change matrix T view to convert the internal parameter view matrix V s into a standard view matrix V a×b in the standard coordinate system;

[0011] S4, obtaining a depth image matrix M a×b of a road in front of the user through the depth camera of the blind guiding device, and performing a binarization operation on the depth image matrix M a×b to obtain a binarization matrix N m×n ;

[0012] S5, obtaining a matrix C a×b through the human body orthographic projection matrix A m×b and the binarization matrix N m×n , performing a traversal scanning operation on the human body orthographic projection matrix A m×b in the matrix C i , defining i as a traversal scanning number, recording a column number X a×bX0, the column number where the center line is located; Y0, the column number where the center line of the passable region is defined. If the scanning is interrupted, the distance difference of the horizontal coordinates δX and the rotation angle δθ are calculated, and X is reset i , the rotation angle is updated by traversing the scanning iteration, and the minimum value of the rotation angle is obtained;

[0013] S6, if there is a passable region, guide the user to move in the corresponding direction; if there is no passable region, appropriately reduce the current threshold distance D and repeat steps S4-S6; if the current threshold distance D is less than the minimum safety distance c1, guide the user to turn.

[0014] Further, in step S1, the number of pixel points occupied by the object or human body in the depth image at different distances is different, according to the similarity principle of optical imaging, the conversion ratio of the actual object length and the corresponding pixel points in the depth image at a certain imaging distance can be calculated, according to the ratio, the relevant information of the user is measured, and the specific processing process is as follows:

[0015] S101, a small cube with side length a is placed in front of the depth camera at a distance s1, and the minimum depth value d1 on the small cube and the number of pixel points occupied by the side length of the cube in the image at this time are recorded by the depth image;

[0016] S102, move the small cube to the front of the depth camera at a distance s2, and record the minimum depth value d2 on the small cube and the number of pixel points occupied by the side length of the cube at this time according to the depth image at this time;

[0017] S103, the minimum depth value and the number of pixel points obtained in steps S101 and S102 are solved to obtain the scaling ratio f(d) of the actual size and the corresponding pixel points in the depth image, wherein f(d)=kd+b, d is the depth value of the obtained depth image;

[0018] S104, the user stands facing the depth camera, and the minimum depth value d0 of the human body in the depth image at this time, the horizontal pixel point number n x , and the vertical pixel point number n y , d0, n x , n y are substituted into the scaling ratio f(d) to obtain the height h1 and the shoulder width w1 of the user.

[0019] Further, the initial threshold distance D1, the current threshold distance D and the minimum safety distance c1 are set as follows:

[0020] The initial threshold distance D1 is set as a distance at which the depth camera can stably image, and the minimum safety distance c1 is set as a limit distance at which the user does not collide with the obstacle when walking considering the human reaction time; wherein the initial threshold distance D1 is an initial value of the current threshold distance D, and the minimum safety distance c1 is a minimum value of the current threshold distance D.

[0021] Further, the human orthographic projection matrix A is set m×n , which is used as a simplification and enlargement of the actual shape of the user's body, and is regarded as the user in subsequent processing, and is specifically set as follows:

[0022] In the determination process, the collision area of the human front is appropriately enlarged according to the scaling ratio f(d) and the current threshold distance D, and the human orthographic projection matrix A is obtained m×n ; wherein m=1.1*h1*f(D), n=1.1*w1*f(D), and the element values in the human orthographic projection matrix A m×n are all initialized to 1.

[0023] Further, in step S3, due to the perspective principle, the image obtained by the depth camera is in the camera coordinate system, in order to identify the distance in the subsequent, it is necessary to transform the object in the camera coordinate system to the standard coordinate system represented by a pixel position (u, v, w), and move the coordinates of the ground to the u-w plane. The whole process can be subdivided into the following steps: the object in the camera coordinate system is converted to the standard coordinate system through model transformation, then the relative position coordinates of the object and the depth camera are obtained through camera transformation, and then the object after camera transformation is projected to [-1,1] 3 above the three-dimensional space through perspective projection transformation, and finally the coordinates of the ground are moved to the u-w plane through viewport transformation, and the specific processing process is as follows:

[0024] S301, define the depth camera view direction as g, the vertical upward direction along the view direction as t, and establish the standard coordinate system (u, v, w), which is defined as follows:

[0025]

[0026] S302, set the depth camera position e as the origin, and define the transformation matrix as V cam , and the formula is as follows:

[0027] wherein x u , x v , x w , y u , y v , y w , z u , zv , z w are the x, y, z components of the object in the original coordinate system under the projection of the standard coordinate system O uvw e , y e , z e is the coordinate position of the depth camera position e; the right matrix represents the first step of moving the depth camera position to the origin, and the left matrix represents the second step of aligning it with the standard coordinate system O uvw .

[0028] S303, define the visual space where the camera is located as follows:

[0029] l≡left plane; r≡right plane;

[0030] b≡top plane; t≡bottom plane;

[0031] n≡near plane; f≡far plane;

[0032] S304, convert the depth camera perspective matrix T view to [-1,1] 3 three-dimensional space, and perform convolution calculation to obtain the orthogonal transformation matrix V ortho , the formula is as follows:

[0033]

[0034] S305, take the viewpoint as the origin of the v-axis and the plane at w=-n as the projection plane, and use the properties of similar triangles and two points to substitute any two points on the near and far planes respectively to obtain the projection transformation matrix V persp→ortho , the formula is as follows:

[0035]

[0036] S306, define the perspective projection transformation matrix V per , the formula is as follows:

[0037] V per = V ortho V persp→ortho ;

[0038] The calculation result is as follows:

[0039]

[0040] S307, convert the depth camera perspective matrix in [-1,1] 3 three-dimensional space to the standard coordinate system to obtain the viewport transformation matrix V view , the formula is as follows:

[0041] ​

[0042] S308. Define the coordinate system transformation matrix T view The formula is as follows:

[0043] T view =V viewport V per V cam ;

[0044] The standard viewpoint matrix V in the standard coordinate system is obtained through calculation. s The formula is as follows:

[0045] V s =T view V in .

[0046] Further, in step S4, a depth image matrix of road obstacles in front of the user is acquired through the depth camera of the guide device. The vertical distance of each pixel in the depth image matrix from the depth camera is calculated using the depth value of each pixel. Simultaneously, the depth image matrix is ​​binarized to project obstacles with a vertical distance less than a current threshold distance onto the projection plane containing the current threshold distance, while filtering out obstacles with a vertical distance greater than the current threshold distance. The specific processing steps are as follows:

[0047] S401. Obtain a depth image matrix M of road obstacles in front of the user through the depth camera of the guide device. a×b Depth image matrix M a×b The value of each pixel in the image represents the distance from the corresponding point on the object to the depth camera, and the horizontal distance z from the object to the camera in the depth image is calculated according to the Pythagorean theorem in three-dimensional space.

[0048] S402, Regarding the depth image matrix M a×b Binarization is performed to obtain the binarized matrix N. a×b Set the z-values ​​that are greater than or equal to the initial threshold distance to 1, and set the horizontal distances z-values ​​that are less than the initial threshold distance to 0.

[0049] Furthermore, in step S5, to avoid the influence of uneven road surface on obstacle judgment, the binarized matrix needs to be segmented. The human body orthographic projection matrix is ​​then scanned through the segmented matrix to obtain the passable regions, and the center line of the passable regions is calculated. The difference in the number of columns between the center line of the passable regions and the center line of the binarized matrix is ​​converted into the actual rotation angle. The matrix traversal is repeated, and the rotation angle is iterated until the matrix traversal ends, obtaining the minimum value of the rotation angle. The specific processing procedure is as follows:

[0050] S501, According to the human body orthographic projection matrix A m×n, take the binary matrix N a×b from the a-1.1m-1 row to the a-0.1m row of the binary matrix N m×n , obtain a matrix C m×b with the same number of rows as the human body orthographic projection matrix A m×n ;

[0051] S502, performing a traversal scanning operation on the human body orthographic projection matrix A m×b , defining i as the traversal scanning times, if the ones matrix is obtained through i consecutive scanning, recording the column number X i of the leftmost column of the human body orthographic projection matrix in the i-th traversal (i=1, 2…n); defining the column number X0 of the middle line in the binary matrix N a×b as X0=b / 2; defining the column number Y0 of the passable area middle line as Y0=(X n -X1+m) / 2; if the scanning is interrupted, calculating the horizontal distance difference between X0 and Y0 defining the initial value of the rotation angle δθ as 180°, calculating the rotation angle δθ through , and resetting X i =0 and i=0; updating the rotation angle through the traversal scanning iteration, obtaining the minimum value |δθ min | of the absolute value of the rotation angle.

[0052] Further, in the step S6, whether there is a passable area in front is judged through the minimum value |δθ min | of the absolute value of the rotation angle, and the user is guided to proceed, and the specific processing process is as follows:

[0053] S601, if |δθ min |≠180°, the user is guided to turn in the direction of |δθ min | degrees relative to the front, wherein when δθ min >0°, the user is guided to turn right; when δθ min <0°, the user is guided to turn left; and when |δθ min |=0°, the user is guided to continue to go straight.

[0054] S602, if |δθ min |=180°, it is judged that there is no passable area in the current route, and then the current threshold distance D is set as D=D-50cm. Whether the current threshold distance D is less than the minimum safety distance c1 is judged: if the current threshold distance is not less than the minimum safety distance c1, the steps S4-S6 are re-executed; if the current threshold distance D is less than the minimum safety distance c1, the user is guided to turn left by 30 degrees, the current threshold distance is initialized as the initial threshold distance, and the steps S2-S6 are re-executed.

[0055] The present application has the following advantages and effects relative to the prior art:

[0056] (1)The blind guiding method disclosed by the application is applicable to all environments in which a depth camera can normally collect a depth image, the collected depth image contains position and distance information of all obstacles within the imaging range of the depth camera, and the obtained human body model is projected into a depth binary image through depth image segmentation and human body model orthographic projection algorithm, which can provide effective obstacle avoidance guidance for all obstacles appearing in the depth binary image, and does not have problems such as limited object type recognition.

[0057] (2)The blind guiding method based on the target detection algorithm has high requirements for the computing power of the blind guiding device used, and it is difficult to deploy a large model on an embedded development board. The blind guiding method disclosed by the application takes a wearable blind guiding device as the hardware basis, obtains a depth image stream in front of the road in real time through a depth camera, and updates the obtained depth image in real time without storing a large amount of data. Compared with the target detection algorithm, the method does not need to recognize a large number of objects, so it does not need to deploy a large target detection model on an embedded development board, which can effectively save hardware computing power and memory consumption, and is suitable for various types of depth cameras and embedded development boards that meet certain computing power requirements.

[0058] (3)The blind guiding method based on the target detection algorithm effectively solves the problem of insufficient computing power of the embedded development board through cloud computing. However, due to the high delay of cloud computing, considering the high real-time and high dynamicity requirements of blind guiding, the blind guiding device using cloud computing cannot ensure the safety of visually impaired people during travel. The blind guiding method disclosed by the application uses a depth camera to collect images in real time, processes each frame of the video stream based on an embedded development board, and determines the blind guiding and obstacle avoidance direction in the current state based on the depth binary image obtained by the current image processing, which has the characteristics of high dynamicity and high real-time.

[0059] (4)Compared with the blind guiding method based on the SLAM algorithm, the blind guiding method disclosed by the application can effectively save the computing power and memory consumption of the embedded device while fully meeting the needs of blind guiding, without 3D modeling and storage of the road, and can provide fast and clear blind guiding information for visually impaired people in a complex road environment with high dynamicity and high real-time. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application. In the drawings:

[0061] Figure 1 is a flowchart of a blind guiding method based on depth image segmentation and human body model orthographic projection algorithm disclosed by the application;

[0062] Figure 2 is a depth map obtained after the depth camera in the scene of embodiment 1 of the present application is collected and processed;

[0063] Figure 3 is a binary image obtained after the depth map shown in Figure 2 is binarized;

[0064] Figure 4 is a binary image obtained after the depth camera in the scene of embodiment 2 of the present application is collected and processed, corresponding to the position information of the user in the turning process through the right turning corner road. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0066] Embodiment 1

[0067] This embodiment will provide a guide blind method based on depth image segmentation and human body model orthographic projection algorithm according to the flow chart shown in Figure 1 . In the scene shown in Figure 2 , there is a door hole with a height of 2.5 m and a width of 1 m 3 m in front of the depth camera, and the two sides of the door hole are obstacles that cannot be crossed. The specific process is as follows:

[0068] S1, a small cube with a side length of 10 cm is placed 100 cm in front of the Orbbec Gemini2 binocular structured light depth camera, and the minimum depth value d1 on the small cube and the pixel point number n1 occupied by the side length of the cube in the image at this time are recorded by the depth image. The small cube is moved back to 300 cm in front of the depth camera, and the minimum depth value d2 on the small cube and the pixel point number n2 occupied by the side length of the cube at this time are recorded according to the depth image at this time. The scaling ratio f(d) of the actual size and the corresponding pixel point number in the depth image is obtained by the minimum depth value and the pixel point number, wherein f(d)=kd+b, d is the depth value of the obtained depth image; the user stands facing the depth camera, and the minimum depth value d0 of the human body in the depth image at this time, the horizontal pixel point number n x , and the vertical pixel point number n y are obtained; d0, n xn y Substituting the scaling factor f(d), we obtain the user's height as 176cm and shoulder width as 46cm.

[0069] S2. Set the initial threshold distance to 300cm, which is the distance at which the depth camera can stably image. Considering human reaction time, set the minimum safe distance to 100cm. Let the current threshold distance be D, where the initial value of the current threshold distance D is 300cm and the minimum value is 100cm. Based on the scaling ratio f(d) and the current threshold distance D, appropriately enlarge the collision area of ​​the human body's front to obtain the human body orthographic projection matrix A. 321×79 The human body orthographic projection matrix A 321×79 All element values ​​in the array are initialized to 1.

[0070] S3. Transfer the image obtained by the depth camera to the standard coordinate system, then obtain the relative position coordinates of the object and the depth camera through camera transformation, and finally project the object after camera transformation onto [-1,1] through perspective projection transformation. 3 In three-dimensional space, the ground coordinates are finally moved to the uw plane through viewport transformation. The specific processing steps are as follows:

[0071] S301. Define the viewing direction of the depth camera as g, and the vertical upward direction along the viewing direction as t. Establish a standard coordinate system (u, v, w) as follows:

[0072] v = w × u,

[0073] S302. Set the depth camera position e as the origin and define the transformation matrix as V. cam The formula is as follows:

[0074] Where, x u x v x w y u y v y w , z u z v z w These represent the x, y, and z components of the object in its original coordinate system and their corresponding values ​​in the standard coordinate system O. uvw Projection below; x e y e z e Here are the coordinates of the depth camera position e; the right-hand matrix represents the first operation of moving the depth camera position to the origin, and the left-hand matrix represents the second operation of aligning it with the standard coordinate system O. uvw coincide.

[0075] S303, Define the camera's visible space as follows:

[0076] l ≡ left plane; r ≡ right plane;

[0077] b≡top surface; t≡bottom surface;

[0078] n≡near plane; f≡far plane;

[0079] S304, Depth camera view matrix T view Convert to [-1,1] 3 In three-dimensional space, convolution is performed to obtain the orthogonal transformation matrix V. ortho The formula is as follows:

[0080]

[0081] S305. Treat the viewpoint as the origin of the v-axis, and the plane at w = -n as the projection plane. Using the properties of similar triangles and the two-point property, substitute any two points on the near and far planes respectively to obtain the projection transformation matrix V. persp→ortho The formula is as follows:

[0082]

[0083] S306, Define the perspective projection transformation matrix V per The formula is as follows:

[0084] V per =V ortho V persp→ortho ;

[0085] The calculation results are as follows:

[0086]

[0087] S307, [-1,1] 3 The viewport transformation matrix V is obtained by transforming the depth camera's viewpoint matrix in 3D space to the standard coordinate system. view The formula is as follows:

[0088]

[0089] S308. Define the coordinate system transformation matrix T view The formula is as follows:

[0090] T view =V viewport V per V cam ;

[0091] The standard viewpoint matrix V in the standard coordinate system is obtained through calculation. sThe formula is as follows:

[0092] V s =T view V in .

[0093] S4. The depth image matrix of the road obstacles in front of the user and the depth value of each pixel in the depth image matrix are obtained through the depth camera of the guide device. The vertical distance of each pixel in the depth image from the depth camera is calculated. At the same time, the depth image matrix is ​​binarized. The specific processing steps are as follows:

[0094] S401. Obtain a depth image matrix M of road obstacles in front of the user through the depth camera of the guide device. 480×640 Depth image matrix M 480×640 The value of each pixel in the image represents the distance from the corresponding point on the object to the depth camera, and the horizontal distance z from the object to the camera in the depth image is calculated according to the Pythagorean theorem in three-dimensional space.

[0095] S402, Regarding the depth image matrix M 480×640 Binarization is performed to obtain the binarized matrix N. 480×640 , obtain Figure 3 The binarized image is shown; z values ​​greater than or equal to the initial threshold distance are set to 1, and horizontal distances z less than the initial threshold distance are set to 0.

[0096] S5. Based on the human body orthographic projection matrix A 321×79 Take the binary matrix N 480×640 Lines 126 to 448 yield the orthographic projection matrix A of the human body. 321×79 Matrix C with the same number of rows 321×640 ; Project the human body orthographic matrix A 321×79 In matrix C 321×640 The process involves a traversal scan operation, where i is the number of traversal scans. If the ones matrix is ​​obtained after i consecutive scans, the column number X of the leftmost column of the human orthographic projection matrix during the i-th traversal is recorded. i (i = 1, 2, ..., 79); Define the binary matrix N. 480×640 The column number X0 containing the midline is defined as X0 = 320; the column number Y0 that can pass through the midline in the region is defined as Y0 = (X... 79 -X1+321) / 2; When the scan is interrupted, calculate the difference in the horizontal coordinate distance between X0 and Y0. pass Calculate the rotation angle δθ and reset X. i =0, i=0; by iteratively updating the rotation angle through traversal scanning, the minimum absolute value of the rotation angle |δθ is obtained. min |=30°,δθ min = -7°;

[0097] S6. Guide the user to move forward 7 degrees to the left and repeat steps S4-S6.

[0098] Example 2

[0099] This embodiment will be based on Figure 4 The flowchart shown illustrates a guide method for the blind based on depth image segmentation and orthographic projection of a human model. Figure 3 In the scenario shown, there is a narrow right-turn road in front of the user. There is an impassable wall 260cm in front of the user's depth camera. There is a road to the right that allows the user to pass safely. The specific procedure for safely guiding the user to make the right turn in this scenario is as follows:

[0100] S1. Place a small cube with a side length of 10cm 100cm in front of the Orbbec Gemini2 binocular structured light depth camera. Record the minimum depth value d1 and the number of pixels n1 occupied by the cube's side length in the depth image at this time. Move the small cube backward to a position 300cm in front of the depth camera. Record the minimum depth value d2 and the number of pixels n2 occupied by the cube's side length in the depth image at this time. Calculate the scaling factor f(d) between the actual size and the number of pixels in the depth image, where f(d) = kd + b. d represents the depth value of the obtained depth image; the user stands facing the depth camera, and the minimum depth value d0 of the human body in the depth image at this time is obtained, and the number of horizontal pixels n is... x The number of vertical pixels n y d0, n x n y Substituting the scaling factor f(d), we obtain the user's height as 176cm and shoulder width as 46cm.

[0101] S2. Set the initial threshold distance to 300cm, which is the distance at which the depth camera can stably image. Considering human reaction time, set the minimum safe distance to 100cm. Let the current threshold distance be D, where the initial value of the current threshold distance D is 300cm and the minimum value is 100cm. Based on the scaling ratio f(d) and the current threshold distance D, appropriately enlarge the collision area of ​​the human body's front to obtain the human body orthographic projection matrix A. 321×79 The human body orthographic projection matrix A 321×79 All element values ​​in the array are initialized to 1.

[0102] S3. Transfer the image obtained by the depth camera to the standard coordinate system, then obtain the relative position coordinates of the object and the depth camera through camera transformation, and finally project the object after camera transformation onto [-1,1] through perspective projection transformation. 3 In three-dimensional space, the ground coordinates are finally moved to the uw plane through viewport transformation. The specific processing steps are as follows:

[0103] S301. Define the viewing direction of the depth camera as g, and the vertical upward direction along the viewing direction as t. Establish a standard coordinate system (u, v, w) as follows:

[0104] v = w × u,

[0105] S302. Set the position of the depth camera e as the origin, and define the transformation matrix as V. cam The formula is as follows:

[0106] Where, x u x v x w y u y v y w , z u z v z w These represent the x, y, and z components of the object in its original coordinate system and their corresponding values ​​in the standard coordinate system O. uvw Projection below; x e y e z e Here are the coordinates of the depth camera position e; the right-hand matrix represents the first operation of moving the depth camera position to the origin, and the left-hand matrix represents the second operation of aligning it with the standard coordinate system O. uvw coincide.

[0107] S303, Define the camera's visible space as follows:

[0108] l ≡ left plane; r ≡ right plane;

[0109] b≡top surface; t≡bottom surface;

[0110] n≡near plane; f≡far plane;

[0111] S304, Depth camera view matrix T view Convert to [-1,1] 3 In three-dimensional space, convolution is performed to obtain the orthogonal transformation matrix V. ortho The formula is as follows:

[0112]

[0113] S305. Treat the viewpoint as the origin of the v-axis, and the plane at w = -n as the projection plane. Using the properties of similar triangles and the two-point property, substitute any two points on the near and far planes respectively to obtain the projection transformation matrix V. persp→ortho The formula is as follows:

[0114] S306, Define the perspective projection transformation matrix V per The formula is as follows:

[0115] V per =V ortho V persp→ortho ;

[0116] The calculation results are as follows:

[0117]

[0118] S307, [-1,1] 3 The viewport transformation matrix V is obtained by transforming the depth camera's viewpoint matrix in 3D space to the standard coordinate system. view The formula is as follows:

[0119]

[0120] S308. Define the coordinate system transformation matrix T view The formula is as follows:

[0121] T view =V viewport V per V cam ;

[0122] The standard viewpoint matrix V in the standard coordinate system is obtained through calculation. s The formula is as follows:

[0123] V s =T view V in .

[0124] S4. The depth image matrix of the road obstacles in front of the user and the depth value of each pixel in the depth image matrix are obtained through the depth camera of the guide device. The vertical distance of each pixel in the depth image from the depth camera is calculated. At the same time, the depth image matrix is ​​binarized. The specific processing steps are as follows:

[0125] S401. Obtain a depth image matrix M of road obstacles in front of the user through the depth camera of the guide device. 480×640 Depth image matrix M 480×640The value of each pixel in the image represents the distance from the corresponding point on the object to the depth camera, and the horizontal distance z from the object to the camera in the depth image is calculated according to the Pythagorean theorem in three-dimensional space.

[0126] S402, Regarding the depth image matrix M 480×640 Binarization is performed to obtain the binarized matrix N. 480×640 , obtain Figure 3 The binarized image is shown; z values ​​greater than or equal to the initial threshold distance are set to 1, and horizontal distances z less than the initial threshold distance are set to 0.

[0127] S5. Based on the human body orthographic projection matrix A 321×79 Take the binary matrix N 480×640 Lines 126 to 448 yield the orthographic projection matrix A of the human body. 321×79 Matrix C with the same number of rows 321×640 ; Project the human body orthographic matrix A 321×79 In matrix C 321×640 The process involves a traversal scan operation, where i is the number of traversal scans. If the ones matrix is ​​obtained after i consecutive scans, the column number X of the leftmost column of the human orthographic projection matrix during the i-th traversal is recorded. i (i = 1, 2, ..., 79); Define the binary matrix N. 480×640 The column number X0 containing the midline is defined as X0 = 320; the column number Y0 that can pass through the midline in the region is defined as Y0 = (X... 79 -X1+321) / 2; When the scan is interrupted, calculate the difference in the horizontal coordinate distance between X0 and Y0. pass Calculate the rotation angle δθ and reset X. i =0, i=0; by iteratively updating the rotation angle through traversal scanning, the minimum absolute value of the rotation angle |δθ is obtained. min If |δθ = 180°, and there is no passable area on the current route, then set the current threshold distance D = 300cm - 50cm = 250cm. If the current threshold distance D = 250cm is not less than the minimum safe distance c1 = 100cm, repeat steps S4-S5, iteratively updating the rotation angle through traversal scanning to obtain the minimum absolute value of the rotation angle |δθ. min |=30°.

[0128] S6. Guide the user to move forward 30 degrees to the right, and repeat steps S4-S6 until the user is safely guided to complete a 90-degree right turn.

[0129] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A guide for the blind method based on depth image segmentation and human model orthographic projection algorithm, characterized in that, The guiding method for the blind comprises the following steps: S1, calibrate the depth camera of the guide dog device with a small cube with side length a * , get the scaling ratio f(d), d is the depth value of the depth image obtained by the depth camera, the user stands facing the depth camera, and the minimum value d0 of the depth value in the human body pixel points, the horizontal pixel width n x and the vertical pixel height n y at this time are obtained, d0, n x , n y are substituted into the scaling ratio f(d) to obtain the height h1 and the shoulder width w1 of the user; S2, set the initial threshold distance D1 and the minimum safety distance c1, set the current threshold distance as D, according to the scaling ratio f(d), through the height h1 and the shoulder width w1 of the user, construct the human body orthographic projection matrix A m×n , m is the pixel number of the human body model in the vertical direction of the matrix, and n is the pixel number of the human body model in the horizontal direction of the matrix. S3, set the standard coordinate system O uvw Wherein u is the horizontal direction, v is the vertical direction, w is the vertical direction, the ground is the u-w plane, the internal parameter view matrix V of the depth camera in Through the coordinate system change matrix T view Rigid transformation, converted into the standard view matrix V in the standard coordinate system s ; S4, obtaining a depth image matrix M of a road in front of the user by a depth camera of the blind guiding device a×b , and performing a binarization operation on the depth image matrix M a×b to obtain a binarization matrix N a×b ; S5, traversing the human body orthographic projection matrix A m×n and the binary matrix N a×b , obtaining the matrix C m×b , traversing the human body orthographic projection matrix A m×n in the matrix C m×b , defining i as the traversal scanning times, if the ones matrix is obtained by continuous i times scanning, recording the column number X of the leftmost column of the human body orthographic projection matrix in the i-th traversal i , defining the column number X0 of the line in the binary matrix N a×b , defining the column number Y0 that can pass through the line of the region, if the scanning is interrupted, calculating the horizontal coordinate distance difference δX and the rotation angle δθ of the two X0 and Y0, and resetting X i , updating the rotation angle by iterative traversal scanning, obtaining the minimum value of the rotation angle; S6, if there is a passable area, guiding the user to move forward in the corresponding direction; if there is no passable area, appropriately reducing the current threshold distance D and repeating steps S4-S6; if the current threshold distance D is less than the minimum safety distance c1, guiding the user to turn.

2. The guiding method based on the depth image segmentation and human model orthographic projection algorithm according to claim 1, characterized in that, The step S1 process is as follows: S101, place a small cube with a side length of a * in front of the depth camera at a distance s1, record the minimum depth value d1 on the small cube in the image at this time and the number of pixel points n1 occupied by the side length of the cube through the depth image. S102, moving the small cube backward to a distance s2 in front of the depth camera, recording the minimum depth value d2 on the small cube at this time and the number of pixel points n2 occupied by the side length of the cube according to the depth image at this time; S103, the minimum depth value obtained in steps S101 and S102 is associated with the number of pixel points to obtain a scaling ratio f(d) of the actual size and the number of pixel points corresponding to it in the depth image, wherein f(d)=kd+b * , d is the depth value of the obtained depth image; S104, the user stands facing the depth camera, and the minimum depth value d0 of the human body in the depth image at this time is obtained, and the horizontal pixel width n x and the vertical pixel height n y are obtained. x , n y are substituted into the scaling ratio f(d) to obtain the height h1 and the shoulder width w1 of the user. 3.The guiding method based on the depth image segmentation and human model orthographic projection algorithm of claim 2, characterized in that, In the step S2, the initial threshold distance D1 is set as a distance at which the depth camera can stably image, and the minimum safety distance c1 is set as a limit distance at which the user does not collide with the obstacle when walking considering the human reaction time; wherein the initial threshold distance D1 is the initial value of the current threshold distance D, and the minimum safety distance c1 is the minimum value of the current threshold distance D.

4. The guiding method based on the depth image segmentation and human model orthographic projection algorithm according to claim 3, characterized in that, In step S2, the collision area of the human body front face is appropriately enlarged according to the scaling ratio f(d) and the current threshold distance D, to obtain a human body front projection matrix A m×n ; Wherein, m = 1.1 * h1 * f (D), n = 1.1 * w1 * f (D), the element values in the human body orthographic projection matrix A m×n are all initialized to 1.

5. The guiding method based on the depth image segmentation and human model orthographic projection algorithm according to claim 4, characterized in that, The step S3 process is as follows: S301, defining the depth camera visual angle direction as g, the vertical upward direction along the visual angle direction as t, establishing a standard coordinate system (u, v, w), and defining as follows: S302, set the depth camera position e as the origin, define the transformation matrix as V cam , the formula is as follows: wherein x u , x v , x w , y u , y v , y w , z u , z v , z w are the projections of the x, y, z components of the object in the original coordinate system into the standard coordinate system O uvw ; x e , y e , z e are the coordinates of the depth camera position e; the right matrix represents the first step of moving the depth camera position to the origin, and the left matrix represents the second step of aligning the depth camera position with the standard coordinate system O uvw ; S303, defining the visual space in which the camera is located as follows: l≡left plane; r≡right plane; f = far plane; S304、Convert the depth camera view matrix to [-1, 1] 3 In the three-dimensional space, the convolution calculation is performed to obtain the orthogonal transformation matrix V ortho The formula is as follows: S305, the viewpoint is regarded as the origin of the v-axis, and the plane at the viewpoint is regarded as the projection plane, similarity triangle properties and two-point properties are used, and any two points on the far and near planes are substituted to obtain a projection transformation matrix V persp→ortho , the formula is as follows:​ S306、define the perspective projection transformation matrix V per The formula is as follows: V per = V ortho V persp→ortho ; The calculation result is as follows: S307、[-1,1] 3 The depth camera view angle matrix in the three-dimensional space is converted into a standard coordinate system to obtain a viewport conversion matrix V viewport , and the formula is as follows: S308, define the coordinate system change matrix T view The formula is as follows: T view = V viewport V per V cam ; The standard view angle matrix V in the standard coordinate system is calculated s The formula is as follows: V s = T view V in .

6. The guiding method based on the depth image segmentation and human model orthographic projection algorithm according to claim 5, characterized in that, The step S4 process is as follows: S401, obtaining a depth image matrix M of a road in front of the user by a depth camera of the blind guiding device a×b , the value of each pixel point in the depth image matrix M a×b represents the distance from the corresponding point on the object to the depth camera, and the horizontal distance z of the object to the camera in the depth image is calculated according to the Pythagorean theorem in three-dimensional space; S402、to the depth image matrix M a×b binaryzation, get binaryzation matrix N a×b ; set z value greater than or equal to initial threshold distance as 1, set horizontal distance z less than initial threshold distance as 0.

7. The guiding method based on the depth image segmentation and human model orthographic projection algorithm according to claim 6, characterized in that, The step S5 process is as follows: S501、According to the human body orthographic projection matrix A m×n , take the a-1.1m-1 row to a-0.1m row of the binary matrix N a×b , get the matrix C m×n with the same number of rows as the human body orthographic projection matrix A m×b ; S502, the human orthographic projection matrix A m×n In the matrix C m×b The traversal scanning operation is performed, define i as the traversal scanning times, if the ones matrix is obtained by continuous i times scanning, record the column number X of the leftmost column of the human orthographic projection matrix in the i-th traversal i ; Define the binary matrix N a×b The column number X0 of the middle line is X0=b / 2; Define the column number Y0 that can pass through the middle line of the region Y0=(X n -X1+m) / 2; If the scanning is interrupted, calculate the horizontal coordinate distance difference between X0 and Y0 Define the initial value of the rotation angle δθ as 180°, calculate the rotation angle δθ through Reset X i =0, i=0; The absolute value of the rotation angle is obtained by traversing the scanning iteration to update the rotation angle, and the minimum value of the absolute value of the rotation angle is |δθ min | 8. The guiding method based on the depth image segmentation and human model orthographic projection algorithm according to claim 7, characterized in that, The step S6 process is as follows: S601, if |δθ min If |≠180°, then guide the user towards |δθ relative to directly in front. min | The direction of | turns, where when δθ min When >0°, guide to turn right; δθ min When the angle is less than 0°, the steering will turn to the left. When |δθ min When | = 0°, guide the user to continue straight; S602, if |δθ min If |δθ| = 180°, it is determined that the current route has no passable region, and the current threshold distance D is set as D = D - 50 cm. It is determined whether the current threshold distance D is less than the minimum safety distance c1. If the current threshold distance is not less than the minimum safety distance c1, steps S4-S6 are re-executed. If the current threshold distance D is less than the minimum safety distance c1, the user is guided to turn left by 30 degrees, and the current threshold distance is initialized as the initial threshold distance. Steps S2-S6 are re-executed.

Images