Pedestrian Navigation Method, Electronic Device and Storage Medium

Abstract

The present invention provides a pedestrian navigation method, including: responding to a navigation request from a user's mobile terminal, and capturing an image of the road where the user is located at preset intervals; determining whether there is at least one first obstacle on the road according to the captured multiple images; identifying pedestrians in the multiple images, and determining the movement trajectory of each pedestrian; judging whether each pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian; when the pedestrian deviates from walking in the single direction, determining that there is at least one second obstacle on the road; and when it is determined that there is the first obstacle and / or the second obstacle on the road, sending an obstacle avoidance prompt message to the mobile terminal. The present invention also provides an electronic device and a storage medium. The present invention improves the detection accuracy of obstacles and optimizes the navigation effect.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous navigation, and in particular, to a pedestrian navigation method, an electronic device, and a storage medium. Background Art

[0002] With the development of science and technology, navigation technology has been widely used in people's daily lives. Existing navigation technologies usually involve navigation of routes between a starting point and a destination. For example, users can use map applications for driving and walking navigation. However, for visually impaired people, although there are means of walking navigation by laying blind paths on the road in the existing technology, there is still a lack of navigation for the actual walking environment, such as detection of obstacles on the road, which causes inconvenience to the daily travel of visually impaired people. Summary of the Invention

[0003] In view of this, it is necessary to provide a pedestrian navigation method, an electronic device, and a storage medium, which can detect obstacles on the road through a camera device and provide navigation for visually impaired people.

[0004] A first aspect of the present invention provides a pedestrian navigation method, the method comprising:

[0005] Responding to a navigation request from a mobile terminal of a user, and capturing an image of the road where the user is located at preset time intervals;

[0006] Judging whether there is at least one first obstacle on the road according to a plurality of captured images;

[0007] Identifying pedestrians in the plurality of images, and determining the movement trajectory of each pedestrian;

[0008] Judging whether each pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian;

[0009] When the pedestrian deviates from walking in the single direction, determining that there is at least one second obstacle on the road; and

[0010] When it is determined that there is the first obstacle and / or the second obstacle on the road, sending an obstacle avoidance prompt message to the mobile terminal.

[0011] Preferably, the method further comprises:

[0012] When it is determined that there is the first obstacle and / or the second obstacle on the road, judging whether the first obstacle and the second obstacle are located on the walking path of the user; and

[0013] When it is determined that the first obstacle and the second obstacle are on the user's walking path, send the obstacle avoidance prompt information to the mobile terminal.

[0014] Preferably, the determining whether there is at least one first obstacle on the road according to the captured multiple images includes:

[0015] Segment the captured multiple images according to the fully convolutional algorithm and the conditional random field algorithm;

[0016] Determine whether the segmented images contain the contours of other objects except the road contour;

[0017] When the segmented images contain the contours of other objects except the road contour, determine that there is the first obstacle on the road; or

[0018] When the segmented images do not contain the contours of other objects except the road contour, determine that there is no first obstacle on the road.

[0019] Preferably, the method further includes:

[0020] When it is determined that there is the first obstacle on the road, identify the category of the first obstacle, wherein the obstacle avoidance prompt information includes the category of the first obstacle.

[0021] Preferably, the identifying the pedestrians in the multiple images and determining the movement trajectory of each pedestrian includes:

[0022] Identify the pedestrians in each image according to the object detection algorithm;

[0023] Mark each pedestrian in each image based on the head; and

[0024] Generate the movement trajectory of each pedestrian according to the change of the position of the head of the pedestrian in the multiple images.

[0025] Preferably, the determining whether the pedestrian maintains a single-direction walking according to the movement trajectory of each pedestrian includes:

[0026] Determine the preset walking path of the pedestrian according to the orientation of the pedestrian in the first image of the captured multiple images;

[0027] Set two threshold lines on both sides of the preset walking path and the head of the pedestrian;

[0028] Select two reference points at the head of the pedestrian in the image;

[0029] Determine whether the two connecting lines between the same reference points in any two of the multiple images intersect the threshold line;

[0030] When it is determined that at least one of the connecting lines between the same reference points in any two of the multiple images intersects the threshold line, determine that the pedestrian deviates from walking in the single direction; or

[0031] When it is determined that neither of the two connecting lines between the same reference points in any two of the multiple images intersects the threshold line, determine that the pedestrian maintains walking in the single direction.

[0032] Preferably, the determining whether the pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian includes:

[0033] Determine the preset walking path of the pedestrian according to the orientation of the pedestrian in the first image among the multiple captured images;

[0034] Set two threshold lines located on both sides of the preset walking path and the head of the pedestrian;

[0035] Select two reference points on the head of the pedestrian in the image;

[0036] Calculate the sum of the first distances and the sum of the second distances between the two reference points and the adjacent threshold lines in the first image and any other image respectively;

[0037] Determine whether the sum of the first distances is less than the sum of the second distances;

[0038] When it is determined that the sum of the first distances is less than the sum of the second distances, determine that the pedestrian maintains walking in the single direction; or

[0039] When it is determined that the sum of the first distances is greater than or equal to the sum of the second distances, determine that the pedestrian deviates from walking in the single direction.

[0040] Preferably, the capturing images of the road where the user is located at every preset time includes:

[0041] When receiving the navigation request, determine the camera device closest to the mobile terminal; and

[0042] Control the camera device to capture images of the road where the user is located at every preset time.

[0043] A second aspect of the present invention provides an electronic device, including:

[0044] A processor; and

[0045] A memory stores a plurality of program modules, and the plurality of program modules are loaded and executed by the processor to perform the above-mentioned pedestrian navigation method.

[0046] The third aspect of the present invention provides a storage medium, on which at least one computer instruction is stored, and the instruction is loaded and executed by the processor to perform the above-mentioned pedestrian navigation method.

[0047] The above-mentioned pedestrian navigation method, electronic device and storage medium can not only detect visible obstacles through a camera device, but also detect obstacles in a blind spot or relatively hidden area by photographing the moving trajectory of pedestrians, effectively improving the detection accuracy of obstacles and optimizing the navigation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings according to the provided drawings without creative efforts.

[0049] Figure 1 It is a schematic diagram of the application environment architecture of the pedestrian navigation method provided by the preferred embodiment of the present invention.

[0050] Figure 2 It is a schematic diagram of the structure of the electronic device provided by the preferred embodiment of the present invention.

[0051] Figure 3 It is a schematic diagram of the structure of the pedestrian navigation system provided by the preferred embodiment of the present invention.

[0052] Figure 4 It is a schematic diagram of the pedestrian moving trajectory provided by the first embodiment of the present invention.

[0053] Figure 5 It is a schematic diagram of the pedestrian moving trajectory provided by the second embodiment of the present invention.

[0054] Figure 6 It is a flowchart of the pedestrian navigation method provided by the preferred embodiment of the present invention.

[0055] MAIN ELEMENT SYMBOL DESCRIPTION

[0056] Electronic device 1

[0057] Processor 10

[0058] Pedestrian navigation system 100

[0059] Selection module 101

[0060] Shooting module 102

[0061] Judgment module 103

[0062] Recognition module 104

[0063] Determination module 105

[0064] Prompt module 106

[0065] Memory 20

[0066] Computer program 30

[0067] Imaging device 40

[0068] Mobile terminal 2

[0069] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. Specific embodiments

[0070] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be described in detail below in conjunction with the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0071] In the following description, many specific details are set forth in order to fully understand the present invention. The described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the present invention.

[0073] Please refer to Figure 1 As shown, it is a schematic diagram of the application environment architecture of the pedestrian navigation method provided by a preferred embodiment of the present invention.

[0074] The pedestrian navigation method in the present invention is applied to the electronic device 1, and the electronic device 1 is communicatively connected to at least one mobile terminal 2 through a network. The network can be a wired network or a wireless network, such as radio, Wireless Fidelity (WIFI), cellular, satellite, broadcast, etc. Among them, the cellular network can be a 4G network or a 5G network.

[0075] The electronic device 1 can be an electronic device installed with a pedestrian navigation program, such as a personal computer, a server, etc. Among them, the server can be a single server, a server cluster, a cloud server, etc.

[0076] The mobile terminal 2 can be a smart phone, a tablet computer, a smart wearable device, etc.

[0077] Please refer to Figure 2 as shown, which is a schematic structural diagram of the electronic device provided by a preferred embodiment of the present invention.

[0078] The electronic device 1 includes, but is not limited to, a processor 10, a memory 20, a computer program 30 stored in the memory 20 and executable on the processor 10, and a plurality of camera devices 40. For example, the computer program 30 is a pedestrian navigation program. When the processor 10 executes the computer program 30, the steps in the pedestrian navigation method are implemented, such as Figure 6 the steps S601 - S606 shown. Or, when the processor 10 executes the computer program 30, the functions of each module / unit in the pedestrian navigation system are implemented, such as Figure 3 the modules 101 - 106 in

[0079] Exemplarily, the computer program 30 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 30 in the electronic device 1. For example, the computer program 30 can be divided into Figure 3 the selection module 101, the shooting module 102, the judgment module 103, the recognition module 104, the determination module 105, and the prompt module 106 in

[0080] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1, and does not constitute a limitation on the electronic device 1. It may include more or fewer components than shown, or combine some components, or different components. For example, the electronic device 1 may further include input / output devices, network access devices, a bus, etc.

[0081] The so-called processor 10 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor 10 may also be any conventional processor, etc. The processor 10 is the control center of the electronic device 1, and connects various parts of the entire electronic device 1 through various interfaces and circuits.

[0082] The memory 20 can be used to store the computer program 30 and / or modules / units. The processor 10 realizes various functions of the electronic device 1 by running or executing the computer program and / or modules / units stored in the memory 20, and by calling the data stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device 1 (such as audio data, phone book, etc.). In addition, the memory 20 may include volatile memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, at least one magnetic disk storage device, flash memory device, or other storage devices.

[0083] In this embodiment, the imaging device 40 is a camera, which is installed near the road and is used to capture images of the road.

[0084] Please refer to Figure 3 as shown in the functional module diagram of the pedestrian navigation system provided by the preferred embodiment of the present invention.

[0085] In some embodiments, the pedestrian navigation system 100 runs in the electronic device 1. The pedestrian navigation system 100 may include multiple functional modules composed of program code segments. The program code of each program segment in the pedestrian navigation system 100 can be stored in the memory 20 of the electronic device 1 and executed by the at least one processor 10 to implement the pedestrian navigation function.

[0086] In this embodiment, the pedestrian navigation system 100 can be divided into a plurality of functional modules according to the functions it performs. Figure 3 As shown, the functional modules may include a selection module 101, a shooting module 102, a judgment module 103, an identification module 104, a determination module 105 and a prompt module 106. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in the memory 20. It can be understood that in other embodiments, the above modules may also be program instructions or firmware solidified in the processor 10.

[0087] The selection module 101 is used to determine the camera device 40 closest to the mobile terminal 2 when the electronic device 1 receives a navigation request from the mobile terminal 2 .

[0088] In this embodiment, when the electronic device 1 receives a navigation request from the mobile terminal 2, it determines the geographical location information of the mobile terminal 2, calculates the distances between the multiple cameras 40 and the mobile terminal 2 based on the geographical location information, and then determines the camera 40 closest to the mobile terminal 2 based on the calculated distances.

[0089] In other embodiments, when the electronic device 1 receives a navigation request from the mobile terminal 2, it determines the geographic location information of the mobile terminal 2, determines the road where the mobile terminal 2 is located based on the geographic location information, determines at least one camera device 40 located on the road based on the road where the mobile terminal 2 is located, calculates the distance between the at least one camera device 40 and the mobile terminal 2, and then determines the camera device 40 closest to the mobile terminal 2 based on the calculated distance.

[0090] The photographing module 102 is used to respond to the navigation request from the user's mobile terminal 2 and photograph the image of the road where the user is located at every preset time.

[0091] In this embodiment, the camera module 102 controls the camera device 40 closest to the mobile terminal 2 to capture the image of the road where the user is located at every preset time. In this embodiment, the preset time is 0.5 seconds. In other embodiments, the preset time can also be set to other appropriate time according to demand.

[0092] The determination module 103 is used to determine whether there is at least one first obstacle on the road according to the multiple captured images.

[0093] In this embodiment, the judgment module 103 first segments the captured multiple images according to the Fully Convolutional Networks (FCN) algorithm and the Conditional Random Field (CRF) algorithm.

[0094] Specifically, the judgment module 103 normalizes the captured images, then inputs them into the FCN network. After multiple convolutional and max-pooling processes, multiple eigenvalues are obtained. The width and height of the output image are 1 / 32 of the initial image. The eigenvalues are upsampled to obtain corresponding upsampled features. Each point in the upsampled features is input into the softmax prediction function to obtain the segmentation map corresponding to the image. Then, the judgment module 103 inputs the segmentation map into the CRF model to optimize the segmentation map. In this embodiment, the segmentation map includes the contours of each object in the image.

[0095] In this embodiment, the judgment module 103 further determines whether the segmented image contains the contours of other objects besides the road contour. Specifically, the judgment module 103 identifies and determines whether the segmentation map contains the contours of other objects besides the road contour according to the contour features.

[0096] In this embodiment, when the judgment module 103 determines that the segmented image contains the contours of other objects besides the road contour, it is determined that there is the first obstacle on the road. When the judgment module 103 determines that the segmented image does not contain the contours of other objects besides the road contour, it is determined that there is no first obstacle on the road. In this embodiment, the first obstacle is an obvious obstacle.

[0097] The recognition module 104 is used to recognize the category of the first obstacle when the judgment module 103 determines that there is the first obstacle on the road.

[0098] In this embodiment, the category is the name of the first obstacle, such as street lamp, billboard, transformer box, bus stop sign, etc.

[0099] The recognition module 104 is also used to recognize the pedestrians in the multiple images and determine the movement trajectory of each pedestrian.

[0100] In this embodiment, the recognition module 104 identifies pedestrians in each image according to the object detection algorithm. Preferably, the object detection algorithm is the MobileNet-SSD model, where the MobileNet-SSD model is a pre-trained model. The recognition module 104 inputs the multiple images into the MobileNet-SSD model, and then can identify pedestrians in each image. In other embodiments, the object detection algorithm can also be the YOLOv3 model.

[0101] In this embodiment, the recognition module 104 further marks each pedestrian in each image based on the head, and generates the movement trajectory of each pedestrian according to the change in the position of the head of the pedestrian in the multiple images.

[0102] The judgment module 103 is further configured to judge whether the pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian.

[0103] Please refer to Figure 4 As shown, in the first embodiment, the judgment module 103 determines the preset walking path of the pedestrian according to the orientation of the pedestrian in the first captured image, and sets two threshold lines on both sides of the preset walking path and the head of the pedestrian. The two threshold lines are set based on the head position of the pedestrian, where one threshold line is near the top of the head and the other threshold line is near the bottom of the head. Figure 4 In the figure, a square box is used to indicate the pedestrian's head, an arrow is used to indicate the pedestrian's preset walking path, and two solid lines are used to indicate the threshold lines.

[0104] The judgment module 103 further selects two reference points at the head of the pedestrian in the image, such as Figure 4 reference points A and B in the figure. The judgment module 103 further judges whether the two connecting lines between the same reference points in any two of the multiple images intersect the threshold line, that is, the judgment module 103 judges Figure 4 whether the connecting line between the two reference points A and the connecting line between the two reference points B in the figure intersect the threshold line. When it is determined that at least one of the connecting lines between the same reference points in any two of the multiple images intersects the threshold line, the judgment module 103 determines that the pedestrian deviates from walking in the single direction. When it is determined that both of the connecting lines between the same reference points in any two of the multiple images do not intersect the threshold line, the judgment module 103 determines that the pedestrian maintains walking in the single direction.

[0105] Please refer to Figure 5As shown, in the second embodiment, the determination module 103 determines a preset walking path of the pedestrian according to the orientation of the pedestrian in the first captured image, and sets two threshold lines located on both sides of the preset walking path and the head of the pedestrian. The two threshold lines are set based on the head position of the pedestrian, wherein one threshold line is near the top of the head and the other threshold line is near the bottom of the head. Figure 5 In the figure, the head of the pedestrian is schematically shown in a square, the preset walking path of the pedestrian is schematically shown by an arrow parallel to the threshold line, and the two solid lines schematically show the threshold lines.

[0106] The determination module 103 further selects two reference points on the head of the pedestrian in the image, for example Figure 5 the reference points A and B in the figure. The determination module 103 further calculates the sum of the first distances and the sum of the second distances between the two reference points and the adjacent threshold lines in the first image and any other image among the multiple images. In this embodiment, the formulas for the determination module 103 to calculate the sum of the first distances and the sum of the second distances are: where the sum of distances is 2d = d 1 + d 2 , ax + by + c = 0 is the linear equation of the threshold line, and (x 1 , y 1 ) is the coordinate of A or B.

[0107] Specifically, the determination module 103 determines whether the pedestrian is moving away from or approaching the imaging device 40 according to the image. For example, when the determination module 103 determines that the image contains the face of the pedestrian, it is determined that the pedestrian is approaching the imaging device 40. When the determination module 103 determines that the image does not contain the face of the pedestrian, it is determined that the pedestrian is moving away from the imaging device 40. When the determination module 103 determines that the pedestrian is moving away from the imaging device 40, it determines whether the sum of the first distances is less than the sum of the second distances. When the sum of the first distances is less than the sum of the second distances, the determination module 103 determines that the pedestrian maintains walking in the single direction. When the sum of the first distances is greater than or equal to the sum of the second distances, the determination module 103 determines that the pedestrian deviates from walking in the single direction.

[0108] When the determination module 103 determines that the pedestrian is approaching the imaging device 40, it determines whether the sum of the first distances is greater than the sum of the second distances. When the sum of the first distances is greater than the sum of the second distances, the determination module 103 determines that the pedestrian maintains walking in the single direction. When the sum of the first distances is less than or equal to the sum of the second distances, the determination module 103 determines that the pedestrian deviates from walking in the single direction.

[0109] The determining module 105 is configured to determine that there is at least one second obstacle on the road when the pedestrian deviates from walking in the single direction. The determining module 105 is further configured to determine that there is no such second obstacle on the road when the pedestrian maintains walking in the single direction. In this embodiment, the second obstacle is a concealed obstacle, such as a pothole or the like.

[0110] The prompting module 106 is configured to send an obstacle avoidance prompt message to the mobile terminal 2 when it is determined that there is the first obstacle and the second obstacle on the road.

[0111] In this embodiment, the obstacle avoidance prompt message includes the category of the first obstacle, and the positions of the first obstacle and the second obstacle relative to the mobile terminal 2, i.e., the user.

[0112] Further, when it is determined that there is the first obstacle and / or the second obstacle on the road, the judging module 103 judges whether the first obstacle and / or the second obstacle is located on the walking path of the user. When it is determined that the first obstacle and / or the second obstacle is located on the walking path of the user, the obstacle avoidance prompt message is sent to the mobile terminal 2.

[0113] Please refer to Figure 6 As shown, it is a flowchart of a pedestrian navigation method provided by a preferred embodiment of the present invention. According to different requirements, the order of steps in the flowchart can be changed, and some steps can be omitted.

[0114] Step S601: In response to the navigation request from the mobile terminal 2 of the user, capture an image of the road where the user is located at preset intervals.

[0115] In this embodiment, step S601 specifically includes determining the camera device 40 closest to the mobile terminal 2 when the electronic device 1 receives the navigation request from the mobile terminal 2.

[0116] In this embodiment, when the electronic device 1 receives the navigation request from the mobile terminal 2, the geographical location information of the mobile terminal 2 is determined, the distances between the plurality of camera devices 40 and the mobile terminal 2 are calculated respectively according to the geographical location information, and then the camera device 40 closest to the mobile terminal 2 is determined according to the calculated distances.

[0117] In other embodiments, when the electronic device 1 receives a navigation request from the mobile terminal 2, it determines the geographical location information of the mobile terminal 2, determines the road where the mobile terminal 2 is located according to the geographical location information, determines at least one camera device 40 located on the road according to the road where the mobile terminal 2 is located, calculates the distances between the at least one camera device 40 and the mobile terminal 2 respectively, and then determines the camera device 40 closest to the mobile terminal 2 according to the calculated distances.

[0118] In this embodiment, the step S601 further includes controlling the camera device 40 closest to the mobile terminal 2 to capture images of the road where the user is located every preset time. In this embodiment, the preset time is 0.5 seconds. In other embodiments, the preset time can also be set to other appropriate times according to requirements.

[0119] Step S602, determining whether there is at least one first obstacle on the road according to the captured multiple images.

[0120] In this embodiment, first, the captured multiple images are segmented according to the Fully Convolutional Networks (FCN) algorithm and the Conditional Random Fields (CRF) algorithm.

[0121] Specifically, the captured images are normalized and then input into the FCN network. After multiple convolutional and max-pooling processes, multiple eigenvalues are obtained. The width and height of the output image are 1 / 32 of the initial image. The eigenvalues are upsampled to obtain corresponding upsampled features. Each point in the upsampled features is input into the softmax prediction function to obtain the segmentation map corresponding to the image. Then, the segmentation map is input into the CRF model to optimize the segmentation map. In this embodiment, the segmentation map includes the contours of each object in the image.

[0122] In this embodiment, it is further determined whether the segmented image contains the contours of other objects except the road contour. Specifically, it is determined whether the segmentation map contains the contours of other objects except the road contour according to the contour features.

[0123] In this embodiment, when it is determined that the segmented image contains the contours of other objects except the road contour, it is determined that there is the first obstacle on the road. When it is determined that the segmented image does not contain the contours of other objects except the road contour, it is determined that there is no first obstacle on the road. In this embodiment, the first obstacle is an obvious obstacle.

[0124] The step S602 may further include identifying the category of the at least one first obstacle when it is determined that the at least one first obstacle exists on the road.

[0125] In this embodiment, the category is the name of the first obstacle, such as street lamp, billboard, transformer box, bus stop sign, etc.

[0126] Step S603, identify pedestrians in the multiple images and determine the movement trajectory of each pedestrian.

[0127] In this embodiment, pedestrians in each image are identified according to the object detection algorithm. Preferably, the object detection algorithm is the MobileNet-SSD model, where the MobileNet-SSD model is a pre-trained model. Inputting the multiple images into the MobileNet-SSD model can identify pedestrians in each image. In other embodiments, the object detection algorithm can also be the YOLOv3 model.

[0128] In this embodiment, each pedestrian in each image is further marked based on the head, and the movement trajectory of each pedestrian is generated according to the change in the position of the head of the pedestrian in the multiple images.

[0129] Step S604, determine whether the pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian.

[0130] As Figure 4 shown, in the first embodiment, the preset walking path of the pedestrian is determined according to the orientation of the pedestrian in the first captured image, and two threshold lines are set on both sides of the preset walking path and the head of the pedestrian. The two threshold lines are set based on the position of the head of the pedestrian, where one threshold line is near the top of the head and the other threshold line is near the bottom of the head. Figure 4 The head of the pedestrian is schematically shown by a square, the preset walking path of the pedestrian is schematically shown by an arrow, and the threshold lines are schematically shown by two solid lines.

[0131] In the first embodiment, two reference points are further selected at the head of the pedestrian in the image, such as Figure 4 reference points A and B in. Further determine whether the two connecting lines between the same reference points in any two of the multiple images intersect the threshold line, that is, determine Figure 4Whether the connecting line between two reference points A and the connecting line between two reference points B intersect the threshold line. When it is determined that at least one connecting line between the same reference points in any two of the multiple images intersects the threshold line, it is determined that the pedestrian deviates from walking in the single direction. When it is determined that neither of the two connecting lines between the same reference points in any two of the multiple images intersects the threshold line, it is determined that the pedestrian maintains walking in the single direction.

[0132] As Figure 5 shown, in the second embodiment, the preset walking path of the pedestrian is determined according to the orientation of the pedestrian in the first captured image, and two threshold lines are set on both sides of the preset walking path and the head of the pedestrian. The two threshold lines are set based on the head position of the pedestrian, wherein one threshold line is near the top of the head and the other threshold line is near the bottom of the head. Figure 5 In which, a square box is used to indicate the head of the pedestrian, an arrow parallel to the threshold line is used to indicate the preset walking path of the pedestrian, and two solid lines are used to indicate the threshold lines.

[0133] In the second embodiment, two reference points are further selected on the head of the pedestrian in the image, such as Figure 5 reference points A and B in. Further, the first distance sum and the second distance sum between the two reference points and the adjacent threshold lines in the first image and any other image among the multiple images are respectively calculated. In this embodiment, the formulas for calculating the first distance sum and the second distance sum are: where the distance sum is 2d = d 1 + d 2 , ax + by + c = 0 is the linear equation of the threshold line, and (x 1 , y 1 ) is the coordinate of A or B.

[0134] Specifically, it is determined according to the image whether the pedestrian is far from or close to the camera device 40. For example, when it is determined that the image contains the face of the pedestrian, it is determined that the pedestrian is close to the camera device 40. When it is determined that the image does not contain the face of the pedestrian, it is determined that the pedestrian is far from the camera device 40. When it is determined that the pedestrian is far from the camera device 40, it is determined whether the first distance sum is less than the second distance sum. When the first distance sum is less than the second distance sum, it is determined that the pedestrian maintains walking in the single direction. When the first distance sum is greater than or equal to the second distance sum, it is determined that the pedestrian deviates from walking in the single direction.

[0135] When it is determined that the pedestrian is approaching the imaging device 40, determine whether the sum of the first distances is greater than the sum of the second distances. When the sum of the first distances is greater than the sum of the second distances, determine that the pedestrian maintains walking in the single direction. When the sum of the first distances is less than or equal to the sum of the second distances, determine that the pedestrian deviates from walking in the single direction.

[0136] Step S605, when the pedestrian deviates from walking in the single direction, determine that there is at least one second obstacle on the road.

[0137] In this embodiment, the second obstacle is a concealed obstacle, such as a pothole or the like.

[0138] Step S606, when it is determined that there is the first obstacle and / or the second obstacle on the road, send an obstacle avoidance prompt message to the mobile terminal 2.

[0139] In this embodiment, the obstacle avoidance prompt message includes the category of the first obstacle, and the positions of the first obstacle and the second obstacle relative to the mobile terminal 2, i.e., the user.

[0140] In this embodiment, when it is determined that there is the first obstacle and / or the second obstacle on the road, determine whether the first obstacle and / or the second obstacle is located on the user's walking path. When it is determined that the first obstacle and / or the second obstacle is located on the user's walking path, send the obstacle avoidance prompt message to the mobile terminal 2.

[0141] If the modules / units integrated in the electronic device 1 are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0142] The pedestrian navigation method, electronic device, and storage medium provided by the present invention can not only detect visible obstacles through a camera device, but also detect obstacles in blind spots or relatively concealed areas by photographing the movement trajectory of pedestrians, effectively improving the detection accuracy of obstacles and optimizing the navigation effect.

[0143] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be construed as limiting the claimed rights. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described in the apparatus claims can also be implemented by the same unit or device through software or hardware. First, second, etc. are used to denote names and do not denote any particular order.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A pedestrian navigation method, characterized in that, the method includes: responding to a navigation request from a user's mobile terminal, and controlling a camera device to capture images of the road where the user is located at preset time intervals; judging whether there is at least one first obstacle on the road according to the captured multiple images; identifying pedestrians in the multiple images and determining the movement trajectory of each pedestrian; judging whether the pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian, including: determining a preset walking path of the pedestrian according to the orientation of the pedestrian in the first image of the captured multiple images, setting two threshold lines on both sides of the preset walking path and the head of the pedestrian, and selecting two reference points at the head of the pedestrian in the image; judging whether two connecting lines between the same reference points in any two images of the multiple images intersect with the threshold lines. When it is determined that at least one of the connecting lines between the same reference points in any two images of the multiple images intersects with the threshold lines, it is determined that the pedestrian deviates from walking in the single direction; or when it is determined according to the multiple images that the pedestrian is far away from the camera device, respectively calculating a first distance sum and a second distance sum between the two reference points and the adjacent threshold lines in the first image and any other image. When it is determined that the first distance sum is greater than or equal to the second distance sum, it is determined that the pedestrian deviates from walking in the single direction; when the pedestrian deviates from walking in the single direction, determining that there is at least one second obstacle on the road; and when it is determined that there is the first obstacle and / or the second obstacle on the road, sending an obstacle avoidance prompt message to the mobile terminal.

2. The pedestrian navigation method according to claim 1, characterized in that, the method further includes: when it is determined that there is the first obstacle and / or the second obstacle on the road, judging whether the first obstacle and the second obstacle are located on the walking path of the user; and when it is determined that the first obstacle and the second obstacle are located on the walking path of the user, sending the obstacle avoidance prompt message to the mobile terminal.

3. The pedestrian navigation method according to claim 1, characterized in that, the judging whether there is at least one first obstacle on the road according to the captured multiple images includes: segmenting the captured multiple images according to the fully convolutional algorithm and the conditional random field algorithm; judging whether the segmented image contains the contour of other objects except the road contour; when the segmented image contains the contour of other objects except the road contour, determining that there is the first obstacle on the road; or when the segmented image does not contain the contour of other objects except the road contour, determining that there is no first obstacle on the road.

4. The pedestrian navigation method according to claim 3, characterized in that, the method further includes: when it is determined that there is the first obstacle on the road, identifying the category of the first obstacle, wherein the obstacle avoidance prompt message includes the category of the first obstacle.

5. The pedestrian navigation method according to claim 1, characterized in that, identifying the pedestrians in the plurality of images and determining the movement trajectory of each pedestrian includes: identifying the pedestrians in each image according to an object detection algorithm; marking each pedestrian in each image based on the head; and generating the movement trajectory of each pedestrian according to the change in the position of the head of the pedestrian in the plurality of images.

6. The pedestrian navigation method according to claim 1, characterized in that, judging whether the pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian further includes: when it is determined that none of the two connecting lines between the same reference points in any two of the plurality of images intersect the threshold line, determining that the pedestrian maintains walking in the single direction.

7. The pedestrian navigation method according to claim 1, characterized in that: judging whether the pedestrian maintains walking in a single direction according to the movement trajectory of each pedestrian further includes: when it is determined that the sum of the first distances is less than the sum of the second distances, determining that the pedestrian maintains walking in the single direction.

8. The pedestrian navigation method according to claim 1, characterized in that, controlling the imaging device to capture images of the road where the user is located at preset time intervals includes: when receiving the navigation request, determining the imaging device closest to the mobile terminal; and controlling the imaging device to capture images of the road where the user is located at the preset time intervals.

9. An electronic device, characterized in that, the electronic device includes: a processor; and a memory in which a plurality of program modules are stored, and the plurality of program modules are loaded and executed by the processor to perform the pedestrian navigation method according to any one of claims 1 to 8.

10. A storage medium having at least one computer instruction stored thereon, characterized in that, the instruction is loaded and executed by the processor to perform the pedestrian navigation method according to any one of claims 1 to 8.

Images