A method for assisting the blind to hail a taxi based on computer vision technology

Abstract

The present invention discloses a method for assisting the blind to take a taxi based on computer vision technology, which obtains the current user's road conditions through the websocket protocol and WeChat applet; obtains the vehicle coordinates by computer vision technology to track the vehicle and calibrates a unique ID for each vehicle; obtains the license plate coordinates of a certain number of frames by computer vision technology to calculate the vehicle height; obtains the distance between the person and the vehicle and the driving speed of the vehicle by the vehicle tracking frame and the principle of similar triangles of a monocular camera; selects the vehicle tracking frame that needs to perform license plate detection and license plate recognition according to the driving speed of the vehicle; obtains the coordinates and license plate number of the license plate by OCR technology to find the target vehicle; and feeds back the vehicle information corresponding to the target vehicle ID to the user through the websocket protocol. The present invention solves the problems of insufficient real-time performance of the prior art, the obstruction of the license plate, the inaccurate recognition of the new energy license plate, and the lack of software for assisting the blind to take a taxi.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a method for assisting the blind to hail a taxi based on computer vision technology. Background Art

[0002] Online ride-hailing is a popular way for modern people to travel, but visually impaired users cannot determine the location of online ride-hailing cars because they cannot obtain outdoor information. They need to describe the pick-up point and their own clothes to the driver and go to find the car by themselves, which has a high communication cost. Especially under smooth road conditions, the traffic volume is large, and the probability of license plates being photographed is reduced. The existing detection algorithm needs to detect the license plates of all vehicles, resulting in low real-time performance and the possibility of missing the target vehicle. In addition, the recognition of new energy vehicle license plates is inaccurate. Summary of the invention

[0003] The purpose of the present invention is to provide a method for assisting the blind to hail a taxi based on computer vision technology, in view of the lack of mobile phone software that can help visually impaired users quickly find the target vehicle for reservation in the prior art, and the shortcomings of insufficient real-time detection and recognition algorithms under smooth road conditions, obstruction of license plates, and inaccurate recognition of new energy vehicle license plates. The method of the present invention is deployed on the WeChat applet platform.

[0004] The objective of the present invention is achieved through the following technical solution: A method for assisting the blind to hail a taxi based on computer vision technology, comprising:

[0005] S110: Obtain the current user traffic conditions through the websocket protocol and WeChat applet;

[0006] S120: Using computer vision technology to obtain vehicle coordinates for vehicle tracking and calibrating a unique ID for each vehicle;

[0007] S130: using computer vision technology to obtain a certain number of frames of license plate coordinates to calculate the vehicle height;

[0008] S140: The distance between the person and the vehicle and the driving speed of the vehicle are obtained through the vehicle tracking frame and the similar triangle principle of the monocular camera.

[0009] S150: Filter out a vehicle tracking frame that needs to perform license plate detection and license plate recognition according to the driving speed of the vehicle.

[0010] S160: Obtain the coordinates and license plate number of the license plate through OCR technology to find the target vehicle.

[0011] S170: Feedback the vehicle information corresponding to the target vehicle ID to the user via the websocket protocol.

[0012] Furthermore, in step S110, the current user traffic condition is obtained through the websocket protocol and the WeChat applet, specifically including:

[0013] S1101: The WeChat applet front-end and the algorithm server establish a long connection through the websocket protocol to prepare for real-time image transmission.

[0014] S1102: The WeChat applet front end monitors the real-time frame of the mobile phone camera and converts it into a base64 format string and transmits it to the algorithm server.

[0015] S1103: The algorithm server decodes the base64 and converts it into a picture, which is a specific road condition map of the current user.

[0016] Furthermore, in step S120, the vehicle coordinates are obtained by using computer vision technology to track the vehicle and a unique ID is assigned to each vehicle, specifically including:

[0017] S1201: Use the YOLOv5 model pre-trained on the COCO dataset car Perform vehicle detection on the road condition image to be detected, and obtain the coordinate set of n vehicles through confidence screening

[0018] S1202: Use DeepSORT algorithm to Track the vehicle detection frame in the image, assign a unique ID to each vehicle that enters the image, and make it appear with the same ID in the subsequent tracking frame to obtain the vehicle set And according to Segment to obtain the corresponding vehicle subgraph set

[0019] S1203: If the vehicle ID has appeared in the previous frame, the vehicle information of the previous frame will be retained to obtain the vehicle set of the previous frame.

[0020] Further, in step S130, the coordinates of the license plate of a certain number of frames are obtained by using computer vision technology to calculate the vehicle height, which specifically includes:

[0021] S1301: Define the actual vehicle height H car ={(id1,h1)…(id n ,h n )}, where each pair (id x ,h x )∈H car h in x The initial value defaults to the common height of a sedan, 1.4 meters (most online-hailing taxis are sedans).

[0022] S1302: The license plate set on the CCPD dataset is divided into a training set, a validation set, and a test set. The YOLOv5 model is used to train on the dataset to obtain a pre-trained model f that can extract license plate information. lp .

[0023] S1303: Vehicle subgraph set If conditions permit, license plate detection is performed to obtain a collection of license plate detections Corresponding to the license plate sub-image set that needs to be recognized after license plate detection The condition is: for the vehicle subgraph If the corresponding vehicle id x If it is the first K frames detected, license plate detection is performed; otherwise, no detection is performed.

[0024] S1304: Based on the principle of similar triangles of monocular cameras, there is a connection relationship: real vehicle height / image vehicle height = person-vehicle distance (ignoring the person-license plate distance and the person-vehicle distance error) / camera focal length = real license plate height / image license plate height. The formula is obtained through the connection relationship: real vehicle height = real license plate height / image license plate height*image vehicle height. Through the license plate set D and vehicle set The ratio of h in the figure is the ratio of the license plate image height to the vehicle image height. The real license plate height in China is generally 14 cm. According to the formula, the real vehicle height can be obtained and the real vehicle height set H can be updated. car ={(id1,h1)…(id1,h n )}. For each new vehicle that enters the field of view, the vehicle height is updated K times to obtain a relatively stable and accurate value.

[0025] Furthermore, in step S140, the distance between the person and the vehicle and the driving speed of the vehicle are obtained by using the vehicle tracking frame and the principle of similar triangles of the monocular camera, which specifically includes:

[0026] S1401: Use a geometric method based on the principle of similar triangles of a monocular camera to obtain the focal length of the mobile phone camera in the WeChat applet.

[0027] S1402: Based on the actual vehicle height set H car ={(id1,h1)…(id n ,h n )} and vehicle collection Get the ratio of the real height of the vehicle to the height of the vehicle in the picture, and use the height of the vehicle as the real object height into the formula based on the principle of similar triangles of monocular cameras: actual distance between camera and object = (focal length * actual height of object) / picture height, and get the distance set D between people and vehicles car ={(id1,d1)…(idn ,d n )}. (Since the license plate may be blocked by the vehicle in front or other objects, once the actual vehicle height is obtained, there is no need to use the license plate as the actual object height for distance measurement).

[0028] S1403: A geometric method based on the principle of similar triangles of a monocular camera can be used to solve the problem of changes in shooting perspective and obtain the actual distance traveled by the vehicle in the lane direction. The vehicle's travel speed is then calculated by the time difference between two frames received by the background.

[0029] Further, in step S1401, the focal length of the mobile phone camera in the WeChat applet is obtained based on a geometric method based on the principle of similar triangles of a monocular camera, specifically including:

[0030] (1) Use the WeChat applet, place the phone vertically, and take a photo of any vertical object in front of you with a known height.

[0031] (2) Obtain the pixel value of the object from the image transmitted to the backend and convert it into centimeters using dpi (the distortion of mobile phone cameras is very small and basically does not require correction. If you want more accuracy, you can also use a calibration plate to obtain the height of the object in the image).

[0032] (3) Measure the actual distance between the phone and the object.

[0033] (4) According to the formula based on the principle of similar triangles of monocular cameras: focal length = (actual distance between camera and object * image height) / actual height of object, the above values ​​are substituted to obtain the focal length of the camera in the WeChat applet (the reason for using height is that width will lose depth when the camera converts 3D to 2D, resulting in reduced accuracy, and height deformation is smaller).

[0034] Furthermore, in step S1403, a geometric method based on the principle of similar triangles of a monocular camera can solve the problem of changes in shooting perspective to obtain the actual distance traveled by the vehicle in the lane direction, specifically including:

[0035] (1) Based on the distance set D between people and vehicles car ={(id1,d1)…(id n ,d n )}, let the distance between the person and the car in the previous frame be d last , d last The horizontal component at the center of the camera's shooting angle is denoted as b last , the component in the vertical direction at the center of the camera's shooting angle is recorded as a last Similarly, the distance between the person and the vehicle in the current frame is d now , d now The horizontal component at the center of the camera's shooting angle is denoted as b now, the component in the vertical direction at the center of the camera's shooting angle is recorded as a now .

[0036] (2) When both frames are located to the left or right of the midline of the viewing angle, it can be proved based on the principle of geometry that the distance traveled by the vehicle on the road has nothing to do with the shooting angle. The travel distance is

[0037] (3) When one of the two frames is located to the left of the midline of the viewing angle and the other is located to the right of the midline of the viewing angle, it can be proved based on the principle of geometry that the distance traveled by the vehicle on the road has nothing to do with the shooting angle. The travel distance is:

[0038] Further, in step S150, the vehicle tracking frame that needs to perform license plate detection and license plate recognition is screened out according to the vehicle's driving speed, specifically including:

[0039] S1501: When the vehicle speed is higher than a certain threshold, it is considered that the vehicle is traveling too fast and the probability of it being the target vehicle is small. When the vehicle speed is too fast, the license plate is definitely blurred and cannot be recognized. Calling the recognition model again will waste time, so the OCR operation is not performed.

[0040] S1502: Gather vehicles Filter out the vehicle sub-graph set that needs license plate detection and license plate recognition within the normal speed range

[0041] S1503: When the vehicle speed is lower than a certain threshold, it is considered that the vehicle has stopped; if it is a target vehicle, the distance and stop status of the vehicle can be notified to the user, allowing the user to actively search according to the prompt.

[0042] Further, in step S160, the target vehicle is found by obtaining the coordinates and license plate number of the license plate through OCR technology, which specifically includes:

[0043] S1601: For the vehicle subgraph set Each subgraph in Use f lp License plate detection, update license plate collection Thereby updating the license plate sub-image collection At the same time, the method of S1304 is used to calculate the real vehicle height set H car ={(id1,h1)…(id n ,h n )} to update.

[0044] S1602: Train text detection and text recognition models on the large-scale public icdar2013 dataset, and pre-train to obtain f DBnet and f CRNN Model.

[0045] S1603: For the car plate sub-image set Each subgraph in Use f DBnet The model performs text detection and uses f CRNN Perform text recognition on the cut text detection frame to obtain the text content set

[0046] S1604: Here we use a license plate recognition text screening method based on area ratio to filter the text content collection Filter and splice to get the final license plate number result If a license plate number is found that is the same as the target one, the vehicle ID is set as the target ID.

[0047] Furthermore, in step S1604, the license plate recognition text screening method based on area ratio specifically includes:

[0048] (1) Since the province letter and the following 6 characters on the new energy green license plate are far apart, they are sometimes treated as two text detection boxes. In addition, there will be small characters on the green license plate, which will cause an additional detection box to appear.

[0049] (2) By screening the two blocks with the largest detection domain in the text recognition results, the largest area is compared with the second largest area. If the ratio exceeds the threshold t, it indicates that the second largest area is the small characters detected and is directly ignored; if the ratio does not exceed the threshold t, it indicates that the energy green card province letters and the following 6 characters are recognized separately and merged into the final result.

[0050] Furthermore, in step S170, the vehicle information corresponding to the target vehicle ID is fed back to the user through the websocket protocol, specifically including:

[0051] S1701: After finding the target vehicle, there is no need to use license plate recognition. You only need to track the target ID and simultaneously count the vehicle's driving direction, left and right positions in the field of vision, the vehicle's stop status and other information as the result R.

[0052] S1702: The result R is transmitted in the form of a string to the WeChat applet front end through the websocket protocol of the algorithm server. After receiving the result R, the front end interacts with the user in the form of voice.

[0053] The beneficial effects of the present invention are as follows: the geometric method based on the principle of similar triangles of a monocular camera in the present invention can solve the problem of changes in shooting perspective and obtain the actual distance traveled by the vehicle in the direction of the lane, and more accurately obtain the vehicle's speed to judge the vehicle's stop status and feedback to the user. The license plate recognition text screening method based on area ratio in the present invention can solve the problem of identifying new energy license plates. The present invention screens possible target vehicles to reduce the number of license plate detection and license plate recognition, speed up the overall recognition speed, and will not cause the vehicle to be unrecognized due to delays. The present invention provides the status of the target vehicle through the WeChat applet platform, helping visually impaired users solve the difficulty of finding the target online car-hailing vehicle in the last 10m of daily taxi-hailing, and solves the problems of insufficient real-time performance of the existing technology, obstruction of license plates, inaccurate recognition of new energy license plates, and lack of software to help the blind take taxis. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of the method for assisting the blind to hail a taxi based on computer vision technology of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described below in conjunction with the accompanying drawings and embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0056] like Figure 1 As shown, the present invention provides a method for assisting the blind to hail a taxi based on computer vision technology, comprising the following steps:

[0057] S110: Obtain the current user traffic conditions through the websocket protocol and WeChat applet. Including:

[0058] S1101: The WeChat applet front-end and the algorithm server establish a long connection through the websocket protocol to prepare for real-time image (video) transmission.

[0059] S1102: The WeChat applet front end monitors the real-time frame of the mobile phone camera and converts it into a base64 format string and transmits it to the algorithm server.

[0060] S1103: The algorithm server decodes the base64 and converts it into a picture, which is the user's current specific road condition map.

[0061] S120: Use computer vision technology to obtain vehicle coordinates for vehicle tracking and calibrate a unique ID for each vehicle. Including:

[0062] S1201: Use the YOLOv5 model pre-trained on the COCO dataset car , perform vehicle detection on the road condition image to be detected, and obtain the coordinate set of n vehicles through confidence screening Among them, the confidence threshold is set, and the model f that meets the confidence requirement is retained car Predict the results. is the coordinate of the center point of the vehicle x detection frame, is the width and height of the vehicle x detection box, x=1~n.

[0063] S1202: Using DeepSORT algorithm, Track the vehicle detection frame in the image, assign a unique ID to each vehicle that enters the image, and make it appear with the same ID in the subsequent tracking frame to obtain the vehicle set id x is the ID of vehicle x; and according to Segment to obtain the corresponding vehicle subgraph set is the vehicle subgraph of vehicle x.

[0064] S1203: If the vehicle ID has appeared in the previous frame, the vehicle information of the previous frame will be retained to obtain the vehicle set of the previous frame. The direction of the vehicle is obtained according to the change in the center position of the vehicle detection frame between the previous and next two frames.

[0065] S130: Use computer vision technology to obtain the license plate coordinates of a certain number of frames and calculate the vehicle height. Including:

[0066] S1301: Define the actual vehicle height H car ={(id1,h1)…(id n ,h n )}, where each pair (id x ,h x )∈H car The vehicle x height h in x The initial value defaults to the common height of a sedan, 1.4 meters (most online-hailing taxis are sedans).

[0067] S1302: Divide the license plate set on the CCPD dataset into a training set, a validation set, and a test set, and use the YOLOv5 model to train on the dataset to obtain a pre-trained model f that can extract license plate information. lp .

[0068] S1303: Vehicle subgraph set Perform license plate detection conditionally to obtain a license plate set Corresponding to the license plate sub-image set that needs to be recognized after license plate detection n γ ≤n, is the license plate sub-image of vehicle x; is the coordinate of the center point of the license plate detection frame of vehicle x, is the width and height of the license plate detection frame of vehicle x. The condition is: for the vehicle sub-image If the corresponding vehicle id x If it is the first K frames detected, license plate detection is performed on the current frame. That is, if the difference between the frame in which vehicle x is first detected and the current frame is less than K frames, license plate detection is performed on the current frame; otherwise, no detection is performed.

[0069] S1304: Based on the principle of similar triangles of monocular cameras, there is a continuous relationship: real vehicle height / image vehicle height = person-vehicle distance (ignoring the person-license plate distance and the person-vehicle distance error) / camera focal length = real license plate height / image license plate height. The formula is derived from the continuous relationship: real vehicle height = real license plate height / image license plate height*image vehicle height. Through the license plate set D and vehicle set The ratio of h to the corresponding vehicle is obtained by calculating the ratio of the license plate image height to the vehicle image height. The real license plate height in China is generally 14 cm. According to the above formula, the real vehicle height can be obtained and the real vehicle height set H can be updated. car ={(id1,h1)…(id n ,h n )}.

[0070] Each new vehicle entering the field of view is updated K times in height to obtain a relatively stable and accurate vehicle height value. Specifically, according to step S1303, the vehicle entering the field of view is detected in the frame from which it is detected, and the vehicle height is updated once for each license plate detection, so that each vehicle is detected K times in height, and the vehicle height can be updated K times.

[0071] S140: The distance between the person and the vehicle and the driving speed of the vehicle are obtained through the vehicle tracking frame and the similar triangle principle of the monocular camera.

[0072] S1401: Use a geometric method based on the principle of similar triangles of monocular cameras to obtain the focal length of the mobile phone camera in the WeChat applet. Including:

[0073] (1) Use the WeChat applet, place the phone vertically, and take a photo of any vertical object in front of you with a known height.

[0074] (2) Obtain the pixel value of the object from the image uploaded to the backend and convert it to centimeters using dpi to obtain the height of the object. The distortion of mobile phone cameras is very small and basically does not require correction. If you want to be more precise, you can also use a calibration plate to obtain the height of the object in the image.

[0075] (3) Measure the actual distance between the phone and the object.

[0076] (4) According to the formula based on the principle of similar triangles of monocular cameras: focal length = (actual distance between camera and object * image height) / actual height of object. Substituting the above value into the image, we get the focal length of the camera in the WeChat applet. The reason for using height is that width will lose depth when the camera converts 3D into 2D, resulting in reduced accuracy, while height deformation is smaller.

[0077] S1402: Based on the current real vehicle height set H car ={(id1,h1)…(id n ,h n )} and vehicle collection Get the ratio of the real height of the vehicle to the height of the vehicle in the picture, take the height of the vehicle as the real object height, and put it into the formula based on the principle of similar triangles of monocular cameras: actual distance between camera and object = (focal length * actual height of object) / picture height, and get the distance set D between people and vehicles car ={(id1,d1)…(id n ,d n )}. x The distance between the person and vehicle x. Since the license plate may be blocked by the vehicle in front or other objects, once the real vehicle height is obtained, there is no need to use the license plate as the real object height for distance measurement.

[0078] S1403: A geometric method based on the principle of similar triangles of a monocular camera is used to solve the problem of changes in shooting perspective and obtain the actual distance traveled by the vehicle in the lane direction. The vehicle's driving speed is then calculated by the time difference between two frames (the current frame and the previous frame) received by the background. This includes:

[0079] (1) Based on the distance set D between people and vehicles car ={(id1,d1)…(id n ,d n )}, let the distance between the person and the car in the previous frame be d last , d last The horizontal component at the center of the camera's shooting angle is denoted as b last , the component in the vertical direction at the center of the camera's shooting angle is recorded as a last Similarly, the distance between the person and the vehicle in the current frame is d now , d now The horizontal component at the center of the camera's shooting angle is denoted as b now , the component in the vertical direction at the center of the camera's shooting angle is recorded as a now .

[0080] (2) When both frames are located to the left or right of the midline of the viewing angle, it can be proved based on the principle of geometry that the distance traveled by the vehicle on the road has nothing to do with the shooting angle. The travel distance is:

[0081] (3) When one of the two frames is located to the left of the midline of the viewing angle and the other is located to the right of the midline of the viewing angle, it can be proved based on the principle of geometry that the distance traveled by the vehicle on the road has nothing to do with the shooting angle. The travel distance is:

[0082] S150: Filter out the vehicle tracking frame that needs to perform license plate detection and license plate recognition according to the vehicle's driving speed. Including:

[0083] S1501: When the vehicle speed is higher than the preset driving threshold, it is considered that the vehicle is driving too fast and the probability of it being the target vehicle is small. When the vehicle speed is too fast, the license plate must be blurred and unrecognizable. Calling the recognition model again will waste time, so the OCR operation is not performed.

[0084] S1502: Gather vehicles Filter out the vehicle sub-graph set that needs license plate detection and license plate recognition within the normal speed range n β ≤n, is the license plate sub-image of vehicle x; the normal range means that it does not exceed the driving threshold.

[0085] S1503: When the vehicle speed is lower than a preset stop threshold, it is considered that the vehicle has stopped.

[0086] S160: Use OCR technology to obtain the coordinates and license plate number of the license plate and find the target vehicle. Including:

[0087] S1601: For the vehicle subgraph set Each subgraph in Use f lp License plate detection, update license plate collection Thereby updating the license plate sub-image collection n D ≤n; at the same time, the method of step S1304 is used to calculate the real vehicle height set H car ={(id1,h1)…(id n ,h n )} to update.

[0088] S1602: Train text detection and text recognition models in the large-scale public icdar2013 dataset, and pre-train to obtain f DBnet and f CRNN Model.

[0089] S1603: For the current license plate sub-image set Each subgraph in Use f DBnet The model performs text detection and uses f CRNNPerform text recognition on the cut text detection frame to obtain the text content set n T ≥n D .

[0090] S1604: Here we use a license plate recognition text screening method based on area ratio to filter the text content collection Filter and splice to get the final license plate number result include:

[0091] (1) Since the province letter and the following 6 characters on the new energy green license plate are far apart, they are sometimes treated as two text detection boxes. In addition, there will be small characters on the green license plate, which will cause an additional detection box to appear.

[0092] (2) By screening the two blocks with the largest detection domain in the text recognition results, the largest area is compared with the second largest area. If the ratio exceeds the threshold t, it indicates that the second largest area is the small characters detected and is directly ignored; if the ratio does not exceed the threshold t, it indicates that the energy green card province letters and the following 6 characters are recognized separately and merged into the final result.

[0093] S170: Feedback the vehicle information corresponding to the target vehicle ID to the user through the websocket protocol. Including:

[0094] S1701: cyclically execute steps S120 to S160 for each frame of road condition image acquired in real time until the target vehicle is found in a certain frame. If there is a vehicle with the same license plate number as the target vehicle, it means that the target vehicle has been found, and the vehicle ID is the target vehicle ID.

[0095] S1702: After the target vehicle is found, it is no longer necessary to perform license plate recognition. Instead, it is only necessary to track the target ID and simultaneously count the vehicle's driving direction, left and right positions on the midline of the viewing angle, the vehicle's stop state, and other information as the result R. The driving direction of the target vehicle in the stop state is obtained according to step S1203, and the stop state of the target vehicle is determined by the driving speed obtained according to step S1403 and determined according to step S1503.

[0096] S1703: When the target vehicle is found to be stopped in a certain frame, the result R is transmitted to the WeChat applet front end in the form of a string through the websocket protocol of the algorithm server. After receiving the result R, the front end interacts with the user in the form of voice; specifically, the stopped status of the vehicle and the distance between the vehicle and the person are fed back to the user, allowing the user to actively search according to the prompts.

[0097] In summary, the present invention provides a method for assisting the blind to hail a taxi based on computer vision technology. The method obtains the current user's road conditions through a websocket protocol and a WeChat applet; obtains vehicle coordinates by using computer vision technology to track the vehicle and calibrates a unique ID for each vehicle; obtains license plate coordinates of a certain number of frames by using computer vision technology to calculate the vehicle height; obtains the distance between a person and the vehicle and the vehicle's driving speed through a vehicle tracking frame and the principle of similar triangles of a monocular camera; selects a vehicle tracking frame that needs to perform license plate detection and license plate recognition according to the vehicle's driving speed; obtains the coordinates and license plate number of the license plate through OCR technology to find a target vehicle; and feeds back vehicle information corresponding to the target vehicle ID to the user through the websocket protocol. The present invention proposes a geometric method based on the principle of similar triangles of a monocular camera, which can solve the problem of changes in shooting angle of view and obtain the actual distance traveled by the vehicle in the direction of the lane, and more accurately obtain the vehicle's driving speed to judge the vehicle's stopped state and feedback to the user; the present invention proposes a license plate recognition text screening method based on area ratio to solve the recognition problem of new energy license plates; the present invention screens possible target vehicles to reduce the number of license plate detection and license plate recognition, speed up the overall recognition speed, and will not cause the vehicle to be unrecognized due to delays; the algorithm provides the status of the target vehicle through the WeChat applet platform to help visually impaired users solve the difficulty of finding the target online car-hailing vehicle in the last 10 meters in daily taxi-hailing.

[0098] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

[0099] The present invention is not limited to the above-mentioned embodiments. All other embodiments obtained by ordinary technicians in the field without making creative work in the same or similar manner as the above-mentioned embodiments of the present invention are within the protection scope of the patent of this invention.

Claims

1. A method for assisting the blind to hail a taxi based on computer vision technology, characterized in that: The steps include: S110: Obtain current traffic conditions through websocket protocol and WeChat applet; S120: Use computer vision technology to obtain vehicle coordinates for vehicle tracking and calibrate a unique ID for each vehicle; including: using the model f obtained by pre-training the YOLOv5 model car , perform vehicle detection on the road condition images to be detected, and filter them through confidence, retaining the models that meet the confidence requirements f car Prediction results, get the coordinate set of n vehicles Using DeepSORT algorithm Track the vehicle detection frame in the image, assign a unique ID to each vehicle that enters the image, and make it appear with the same ID in the subsequent tracking frame to obtain the vehicle set And according to Segment to obtain the corresponding vehicle sub-graph set α; if the vehicle ID has appeared in the previous frame, retain the vehicle information of the previous frame and obtain the vehicle set of the previous frame The vehicle's driving direction is obtained according to the vehicle position change between the previous and next two frames; S130: Calculate the vehicle height by obtaining the license plate coordinates of a certain number of frames using computer vision technology; including: defining the real vehicle height H car , set the initial value of vehicle height h x ; Use the pre-trained model f that can extract license plate information obtained by pre-training the YOLOv5 model lp ; Conditionally perform license plate detection on the vehicle sub-graph set α to obtain the license plate detection set D, and correspondingly obtain the license plate sub-graph set γ that needs to be recognized after license plate detection; where the condition is: for the vehicle sub-graph If the corresponding vehicle id x If it is the first K frames detected, license plate detection is performed; through the license plate set D and the vehicle set The ratio of the detection frame height of the same vehicle in the image is obtained to obtain the ratio of the license plate image height to the vehicle image height; given the license plate height, the real vehicle height is calculated according to the following formula, and the real vehicle height set H is updated. car ; Each new vehicle that enters the field of view performs K vehicle height updates; Real vehicle height = real license plate height / image license plate height * image vehicle height; S140: deriving the distance between the person and the vehicle and the driving speed of the vehicle through the vehicle tracking frame and the similar triangle principle of the monocular camera; S150: Filtering out a vehicle tracking frame that needs to perform license plate detection and license plate recognition according to the driving speed of the vehicle; S160: Obtain the coordinates and license plate number of the license plate through OCR technology to find the target vehicle; including: for each sub-graph in the vehicle sub-graph set β Use model f lp Perform license plate detection, update the license plate set D, and thus update the license plate sub-graph set γ, and at the same time update the real vehicle height set H car Update; pre-trained text detection model f DBnet , text recognition model f CRNN ; For each subgraph in the license plate subgraph set γ Use f DBnet The model performs text detection and uses f CRNN Perform text recognition on the cut text detection frame to obtain the text content set Among them, the license plate recognition text screening method based on area ratio is used to select the text content collection Filter and splice to get the final license plate number result If a vehicle with the same license plate number as the target is found, the vehicle ID is set as the target vehicle ID; S170: Feedback the vehicle information corresponding to the target vehicle ID to the user via the websocket protocol.

2. The method for assisting the blind to hail a taxi based on computer vision technology as claimed in claim 1, characterized in that: Step S110 includes: the WeChat applet front end and the algorithm server establish a long connection through the websocket protocol to prepare for real-time image transmission; the WeChat applet front end monitors the real-time frame of the mobile phone camera and converts it into a base64 format string and transmits it to the algorithm server; the algorithm server decodes the base64 and converts it into an image to obtain the current specific road condition map.

3. The method for assisting the blind to hail a taxi based on computer vision technology as claimed in claim 1, characterized in that: Step S140 includes: using a geometric method based on the principle of similar triangles of a monocular camera to obtain the focal length of the camera corresponding to the image; according to the real vehicle height set H car and vehicle collection Get the ratio of the real height of the vehicle to the height of the vehicle in the image, take the height of the vehicle as the real object height, and get the distance set D between the person and the vehicle according to the following formula: car ; Based on the geometric method of similar triangle principle of monocular camera, the actual distance traveled by the vehicle in the lane direction is obtained; and the vehicle's speed is calculated by the time difference between two adjacent frames of pictures; The actual distance between the camera and the object = (focal length * actual height of the object) / picture height.

4. The method for assisting the blind to hail a taxi based on computer vision technology as claimed in claim 3, characterized in that: In step S140: A geometric method based on the principle of similar triangles of a monocular camera is used to obtain the focal length of the camera corresponding to the image; including: (1) Take a picture of any vertical object with a known height in front of you and obtain a picture; (2) Obtain the height of the object in the image; (3) Obtain the actual distance between the camera and the object; (4) The camera focal length is calculated as: (actual distance between camera and object * image height) / actual height of object; Based on the geometric method of similar triangle principle of monocular camera, the actual distance traveled by the vehicle in the lane direction is obtained, including: (1) Based on the distance set D between people and vehicles car ; Assume the distance between the person and the car in the previous frame is d last , d last The horizontal component at the center of the camera's shooting angle is denoted as b last , the component in the vertical direction at the center of the camera's shooting angle is recorded as a last ; Similarly, the distance between the person and the car in the current frame is d now , d noe The horizontal component at the center of the camera's shooting angle is denoted as b now , the component in the vertical direction at the center of the camera's shooting angle is recorded as a now ; (2) When both frames are located to the left or right of the center line of the viewing angle, the travel distance is: (3) When one of the two frames is located on the left side of the visual center line and the other is located on the right side of the visual center line, the travel distance is:

5. The method for assisting the blind to hail a taxi based on computer vision technology as claimed in claim 1, characterized in that: Step S150 includes: for the vehicle set Vehicles whose speed is not higher than the preset driving threshold are screened out to obtain a vehicle subgraph set β that requires license plate detection and license plate recognition; when the speed is lower than the preset stopping threshold, the vehicle is considered to have stopped.

6. The method for assisting the blind to hail a taxi based on computer vision technology as claimed in claim 1, characterized in that: In step S160, the license plate recognition text screening method based on area ratio includes: by screening the two blocks with the largest detection domain in the text recognition results, the largest area is compared with the second largest area. If the ratio exceeds a preset threshold, the text detection frame with the second largest area is ignored; if the ratio does not exceed the preset threshold, the two text detection frames are merged into the final result.

7. The method for assisting the blind to hail a taxi based on computer vision technology as claimed in claim 1, characterized in that: Step S170 includes: looping steps S120 to S160 on the real-time road conditions until the target vehicle is found; after the target vehicle is found, there is no need to perform license plate recognition, only the target vehicle ID needs to be tracked, and information including the vehicle's driving direction, the left and right positions of the center line of the viewing angle, and the vehicle's stop status is counted as a result R; the result R is transmitted to the mini program front end in the form of a string through the websocket protocol of the server, and the front end receives the result R and feeds it back to the user in the form of a prompt.

Images