Driving training reminder method based on wearable device and wearable device

Abstract

The present application discloses a driving training reminder method based on a wearable device and a wearable device, the method is applied to a wearable device provided to a user; the method comprises: obtaining a sight line image through an image acquisition module; obtaining current project information, the project information being related to the currently ongoing driving training project; identifying preset features in the sight line image according to the project information; obtaining corresponding operation reminder information based on the recognition result, and performing display of the operation reminder information. The present application can solve the dependence of trainees on coaching teaching during driving training, and the operation reminder made according to the current situation in the trainees' sight line is more timely and accurate than the guidance of traditional coaches, improving the accuracy of the operation reminder, and thus helping to improve the efficiency of trainees' driving training.

Description

Technical Field

[0001] The present application relates to the field of smart devices, and in particular to a driving training reminder method based on a wearable device and a wearable device. Background Art

[0002] In the existing driving training method, the instructor is usually required to follow the car to guide the student in driving operations. That is, the student operates in the driver's seat, and the instructor sits in the passenger seat to observe the student's actions and the driving conditions of the vehicle. When the student encounters problems in a specific training project, the instructor is required to respond on the spot based on the current vehicle driving conditions and the student's actions, and guide the student to operate the vehicle so that it can continue to complete and meet the project requirements.

[0003] However, this method requires the instructor to accompany the students at all times, and because the instructor is not in the main driving position, it is difficult for the instructor to know the current situation in the students' field of vision. The instructor can only rely on experience and on-the-spot adaptability to guide the students to complete the practice items, resulting in relatively delayed guidance for the students. For example, when the students are practicing reversing into a garage, the vehicle may be about to cross the lane line due to operational errors, but usually by the time the instructor finds out, the vehicle has actually crossed the lane line, and the instructor is also difficult to detect the student's wrong action. At this time, the instructor can only guide the student to restore the vehicle posture and then reverse into the garage again. Not only is time wasted, but the students also find it difficult to know where they made mistakes, resulting in the need to repeat the training many times.

[0004] Therefore, not only can students not receive timely reminders after making operating errors, the accuracy of the reminder content under different lines of sight is relatively poor. Students' teaching also relies on the accompaniment of instructors, which not only has high labor costs, but also if the reminder content is inaccurate it will affect the students' learning effects, making it difficult to improve the efficiency of driving training. Summary of the invention

[0005] The present application provides a driving training reminder method based on a wearable device and a wearable device, which can improve the driving training efficiency of students.

[0006] In a first aspect, the present application discloses a driving training reminder method based on a wearable device, wherein the method is applied to a wearable device provided to a user;

[0007] The wearable device includes an image acquisition module, and the image acquisition module is oriented toward the user's line of sight;

[0008] The method comprises:

[0009] Acquire a sight line image through the image acquisition module;

[0010] Acquiring current project information, where the project information is related to the currently ongoing driving training project;

[0011] identifying preset features in the sight line image according to the project information;

[0012] Corresponding operation reminder information is obtained based on the recognition result, and the operation reminder information is displayed.

[0013] Optionally, the preset features include at least vehicle features and lane features;

[0014] The identifying a preset feature in the sight line image according to the project information includes:

[0015] Determine a corresponding preset reminder condition according to the project information, wherein the preset reminder condition includes a preset relative position relationship between the vehicle feature and the lane feature;

[0016] Identifying location information of vehicle features and lane features in the sight line image;

[0017] It is determined whether the relative position relationship between the vehicle feature in the sight line image and the lane feature meets the preset reminder condition.

[0018] Optionally, identifying the position information of the vehicle features and the lane features in the sight line image includes:

[0019] identifying a first vehicle feature in the line of sight image;

[0020] determining a first region of interest relative to a location of the first vehicle feature;

[0021] Identifying location information of a lane feature in the first region of interest, or identifying location information of a lane feature and a second vehicle feature;

[0022] The determining whether the relative position relationship between the vehicle feature in the sight line image and the lane feature meets the preset reminder condition includes:

[0023] Determining whether the relative position relationship between the lane feature and the first vehicle feature meets a preset reminder condition; or

[0024] It is determined whether a relative position relationship between the lane feature and the second vehicle feature meets a preset reminder condition.

[0025] Optionally, the first vehicle feature is a rearview mirror, and the first region of interest is a mirror surface area of ​​the rearview mirror;

[0026] The identifying the position information of the vehicle features and the lane features in the sight line image includes:

[0027] Enlarging the mirror area;

[0028] Identify vehicle features and lane features in the magnified image;

[0029] Obtaining the pixel width of the lane feature and the pixel width between the vehicle feature and the lane feature;

[0030] The relative distance and relative angle between the current lane feature and the vehicle feature are estimated based on the lane feature, the pixel width between the vehicle feature and the lane feature, and the actual width of the lane.

[0031] Optionally, the identifying a preset feature in the sight line image according to the project information includes:

[0032] The Mask R-CNN deep learning algorithm is used to identify the preset features in the line of sight image.

[0033] Optionally, the preset features include at least vehicle features and lane features;

[0034] The identifying a preset feature in the sight line image according to the project information includes:

[0035] obtaining position information of a first lane feature from the sight line image based on digital image detection;

[0036] The location information of the second lane feature is obtained based on the Mask R-CNN deep learning algorithm;

[0037] The position information corresponding to the target lane feature is obtained according to the first lane feature and the second lane feature.

[0038] Optionally, obtaining the position information of the first lane feature from the sight line image based on digital image detection includes:

[0039] identifying a first vehicle feature in the line of sight image;

[0040] determining a first region of interest relative to a location of the first vehicle feature;

[0041] Performing color space conversion on the target image of the first region of interest so that the target image is converted from the RGB color space to the HSV color space;

[0042] The target image after the color space conversion is processed to obtain the lane feature mask of the target image.

[0043] Optionally, obtaining position information corresponding to a target lane feature according to the first lane feature and the second lane feature includes:

[0044] Get the first lane area, the second lane area and the third lane area;

[0045] Performing similarity analysis on the first lane area, the second lane area, and the third lane area to obtain position information corresponding to the target lane feature;

[0046] Among them, the first lane area is the common area of ​​the first lane feature and the second lane feature; the second lane area is the lane area corresponding to the first lane feature and the second lane feature respectively; the third lane area is the lane area corresponding to the lane feature detected in the historical sight image.

[0047] In a second aspect, the present application further discloses a wearable device, the wearable device comprising an image acquisition module, the image acquisition module facing the user's line of sight;

[0048] The wearable device further includes a processor and a memory, wherein the processor is electrically connected to the memory;

[0049] A computer program is stored in the memory, and the processor executes the driving training reminder method based on a wearable device as described in any one of the above embodiments by calling the computer program stored in the memory.

[0050] Optionally, the wearable device is a camera device worn on a human head;

[0051] The image acquisition module is a camera, and the camera is arranged at a position close to the human eye in the camera device.

[0052] Optionally, the camera device is a pair of smart glasses, and the camera is disposed between two side frames of the smart glasses.

[0053] Optionally, the wearable device further includes an image output module, and the image output module is connected to the processor;

[0054] The processor is further configured to execute:

[0055] Obtain driving operation information of the current vehicle;

[0056] Displaying the driving operation information in the sight line image and performing superimposition processing;

[0057] The sight line image after the superposition processing is output to the image output module, so as to output the image through the image output module.

[0058] From the above, it can be seen that the present application recognizes the acquired line of sight images and makes corresponding operation reminders through the wearable device worn by the user, which can solve the students' dependence on the instructor's teaching during driving training. The operation reminders made according to the students' current line of sight are more timely and accurate than the guidance of traditional instructors, and the accuracy of the operation reminders is improved, which is beneficial to improving the students' driving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of the functional modules of a wearable device provided in an embodiment of the present application.

[0060] Figure 2 A flowchart for implementing a driving training reminder method based on a wearable device provided in an embodiment of the present application.

[0061] Figure 3 A flowchart for obtaining operation reminder information provided in an embodiment of the present application.

[0062] Figure 4 A schematic diagram of an application scenario of a driving training reminder method based on a wearable device provided in an embodiment of the present application.

[0063] Figure 5 A flowchart for implementing the method of obtaining the location information of the target lane features provided in an embodiment of the present application.

[0064] Figure 6 A schematic diagram of the structure of a wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present application can be more easily understood by those skilled in the art, thereby making a clearer definition of the protection scope of the present application.

[0066] Example 1

[0067] See also Figure 1 , the figure shows a schematic diagram of the functional modules of the wearable device provided in an embodiment of the present application.

[0068] like Figure 1 As shown, the wearable device 10 may include an image acquisition module 11 for providing to the user to wear. The wearable device 10 can be worn on the head to obtain an image in the direction of the user's line of sight through the image acquisition module 11. In one embodiment, the image acquisition module 11 can be a camera, which can use an image sensor such as CCD, CMOS, etc. to sense and image the external light, and perform image processing through the image acquisition module 11 or the processing element in the wearable device 10 to obtain a line of sight image in the direction of the user's line of sight.

[0069] Furthermore, in order to obtain an image as close as possible to the user's line of sight, the camera can be set near the user's eyes, such as the forehead, ear, etc. The wearable device 10 can be a helmet, glasses, headphones, or other wearable devices 10 fixed on the head.

[0070] Through the wearable device 10, a line of sight image that is as close as possible to the user's line of sight can be obtained, thereby improving the accuracy and reliability of the operation reminder information.

[0071] See also Figure 2 , the figure shows the implementation process of the driving training reminder method based on a wearable device provided in an embodiment of the present application. The method is applied to a wearable device provided to a user, and the wearable device can be such as Figure 1 A wearable device according to any one of the embodiments shown.

[0072] The method includes the following implementation steps:

[0073] 101. Acquire a sight line image through an image acquisition module.

[0074] The sight line image may be a continuous multi-frame image, and the specific image clarity and frame rate may be set based on the performance of the image acquisition module.

[0075] In one embodiment, after the light signal is acquired by the image sensor of the image acquisition module, it can be processed in the image acquisition module or in other processing elements of the wearable device to obtain a line of sight image.

[0076] It can be understood that the image information in the sight line image changes as the user's head moves and the scenery in the sight line direction changes.

[0077] 102. Obtain current project information, where the project information is related to a currently ongoing driving training project.

[0078] The project information may be project information corresponding to test items such as reversing into a garage, parallel parking, etc., and the type of project information may be determined according to the test items.

[0079] The project information includes at least feature information related to the current driving training project, such as feature information corresponding to the relative position relationship between the lane and the side of the car body required for reversing into a garage, and feature information corresponding to the relative position relationship between the front hood of the car and the lane required for turning training, etc. The present application does not limit the specific feature information type, as long as the feature position and operation judgment method required to complete the driving training project can be identified from the line of sight image through the above-mentioned feature information.

[0080] Of course, in addition to this, the project information may also include other information related to the driving training project, such as the project name, vehicle driving information, etc.

[0081] In some implementations, the project information can be preset on the wearable device and directly retrieved, or it can be preset on the driving training-related equipment on the vehicle or on a terminal such as a user's mobile phone, and obtained by communicating with the wearable device and the above-mentioned terminal. The driving training-related equipment on the vehicle or the terminal such as the user's mobile phone can also be connected to the Internet through the wearable device, so that the wearable device can obtain relevant project information from the Internet.

[0082] 103. Identify preset features in the sight image according to the project information.

[0083] The preset feature may be a feature related to the project information. For example, the feature related to reversing into a garage may be a feature of a rearview mirror. By identifying the position of the rearview mirror and the relative position of the side of the vehicle body and the lane line reflected by the rearview mirror, it is determined whether the current reversing into a garage action meets the preset situation. For another example, the hood of the vehicle and the lane line nearby are identified. When the hood and the lane line are identified, the positional relationship between the vehicle and the lane line can be determined based on the positional relationship between the hood and the lane line.

[0084] It is understandable that different project information may have corresponding different preset features, and the preset features may be pre-set according to different project information, and this application does not impose any limitation on this.

[0085] In one embodiment, the recognition of preset features in the line of sight image can be achieved through artificial intelligence technology or deep learning technology. For example, the preset features are pre-trained through a learning algorithm to obtain a corresponding recognition model, and the recognition model is used to perform feature recognition on the line of sight image, so as to determine and obtain the position information of the preset features in the line of sight image through the recognition model. Specifically, the Mask R-CNN deep learning algorithm can be used to recognize the preset features in the line of sight image. The Mask R-CNN algorithm can simultaneously realize multiple tasks such as target classification, target detection, semantic segmentation, and instance segmentation in one network.

[0086] The specific processing process may include the following steps: the input image first passes through the "CNN" network layer for feature extraction. In order to accelerate the execution speed of the network, the lightweight neural network ShuffleNet v1 is used as the convolutional backbone for corresponding optimization. The input image passes through the "CNN" network layer to generate the corresponding feature map; then, a predetermined number of ROIs are set for each point in the feature map to obtain multiple candidate ROIs; then, these candidate ROIs are sent to the RPN network for binary classification (foreground or background) and BB regression, and a part of the candidate ROIs are filtered out. The remaining ROIs are ROIAligned (i.e., the original image and the pixel of the feature map are first matched, and then the feature map and the fixed feature are matched). Finally, these ROIs are branched, one of which generates the result of target detection, that is, the target detection category (probability value) and the outer rectangular box of the target are finally generated through fully connected layers; the other is deconvolution (FCN operation) to generate a segmented image.

[0087] The above Mask R-CNN network can realize target detection and image segmentation. The target detection can detect the vehicle hood, rearview mirror and other related targets; the image segmentation can separate the positions of lane lines, parking space lines and boundary lines in the image.

[0088] The Mask R-CNN deep learning algorithm can not only effectively improve the recognition efficiency and accuracy of preset features, but also reduce the computing and processing pressure of wearable devices.

[0089] 104. Obtain corresponding operation reminder information based on the recognition result, and display the operation reminder information.

[0090] In one embodiment, after the corresponding position information of the preset features in the line of sight image is identified, the relative position of the current vehicle in the training process can be judged based on the position information between the preset features, and the vehicle control method that needs to be executed next can be predicted, so that the corresponding operation reminder information can be obtained for display.

[0091] For example, if vehicle turning training is being conducted, when the relative position between the identified vehicle hood and the lane line changes, and the change causes the distance between the vehicle hood and the lane line to gradually decrease, a turning operation reminder can be issued to the user, allowing the user to operate the vehicle to make corresponding turns to avoid the vehicle exceeding the lane line.

[0092] The display of the operation reminder information can be performed by voice, image display, or a combination of sound and light. Moreover, the display can be performed by hardware on the wearable device or by an external device connected to the wearable device for communication. The specific display method can be determined according to actual needs, and this application does not limit this.

[0093] Users can promptly learn about the driving situation in the current line of sight image through the operation reminder information displayed by wearable devices or external devices, and can perform corresponding operations based on the operation reminder information without the need for guidance from a coach.

[0094] From the above, it can be seen that the present application recognizes the acquired line of sight images and makes corresponding operation reminders through the wearable device worn by the user, which can solve the students' dependence on the instructor's teaching during driving training. The operation reminders made according to the students' current line of sight are more timely and accurate than the guidance of traditional instructors, and the accuracy of the operation reminders is improved, which is beneficial to improving the students' driving training efficiency.

[0095] Example 2

[0096] See also Figure 3 , the figure shows the implementation process of obtaining operation reminder information provided by an embodiment of the present application.

[0097] In one embodiment, the preset features include at least vehicle features and lane features.

[0098] like Figure 3 As shown, the preset features in the sight line image are identified according to the project information, including:

[0099] 201. Determine a corresponding preset reminder condition according to the project information, where the preset reminder condition is related to a relative position relationship between preset vehicle features and lane features.

[0100] Among them, the vehicle characteristics may include any feature of the vehicle body that is related to the position comparison structure for completing the driving training program or habitually adopted, such as the vehicle's rearview mirror, hood, A-pillar, B-pillar, rearview mirror, etc., and may also include other vehicle body structures.

[0101] The lane feature may be the position, color or shape of a lane line, parking space line or boundary line (eg, a corner of a garage), or other features of a lane line.

[0102] It can be understood that the vehicle characteristics and lane characteristics can be determined according to the needs of actual driving training, and this application does not limit their specific characteristics.

[0103] In one embodiment, a corresponding preset reminder condition is determined based on the project information. The parameters in the preset reminder condition may include the position distance, relative angle or relative direction between the vehicle feature and the lane feature. The preset reminder condition may be triggered by detecting the position distance, relative angle or relative direction between the above-mentioned vehicle feature and the lane feature, and determining whether a certain threshold condition is met.

[0104] For example, when the project information includes the project information of the reverse parking project, the corresponding preset reminder condition can be set as whether the distance threshold between the side of the vehicle body and the lane line of the garage is less than value A, or whether the angle threshold between the side of the vehicle body and the lane line of the garage is less than value B. When the above thresholds are met, the corresponding operation reminder can be performed.

[0105] 202. Identify position information of vehicle features and lane features in the sight line image.

[0106] Among them, the recognition model can be used through deep learning to identify the positions of vehicle features and lane features in the line of sight image, and the position information corresponding to the vehicle features and lane features can be obtained.

[0107] In one embodiment, identifying the position information of the vehicle features and the lane features in the sight line image may include steps 2021 , 2022 and 2023 .

[0108] 2021. Identify a first vehicle feature in the sight image.

[0109] Among them, the first vehicle feature can be a vehicle feature that is prioritized for identification based on actual training needs.

[0110] For example, in the project of reversing into a parking space, the rearview mirror is set as the first vehicle feature to locate the position of the rearview mirror. For another example, in the project of turning, the hood is set as the first vehicle feature.

[0111] Of course, the selection of the first vehicle feature may be determined according to the actual needs of the driving training program.

[0112] 2022. Determine a first region of interest relative to a location of a first vehicle feature.

[0113] The first region of interest may be a ROI (Region Of Interest) region under the recognition model, or may be an image region that is locked for priority recognition based on actual training needs.

[0114] For example, in the reverse parking project, in order to better identify the vehicle situation, the mirror area of ​​the rearview mirror can be set as the first area of ​​interest. For another example, in the turning project, in order to identify the lane line in front of the vehicle, the area above the hood can be set as the first area of ​​interest. The first area of ​​interest can be a pre-set image area located at a specific position of the first vehicle feature.

[0115] Of course, the selection of the first region of interest may also be determined according to the actual needs of the driving training program.

[0116] 2023. Identify the location information of the lane feature in the first region of interest, or identify the location information of the lane feature and the second vehicle feature.

[0117] Among them, other vehicle features relative to the first vehicle feature in the first interest can be defined as the second vehicle feature. For example, in the reverse parking project, the identified rearview mirror is the first vehicle feature, and the side part of the vehicle body reflected in the mirror area of ​​the rearview mirror can be defined as the second vehicle feature, and the corresponding operation reminder is performed by judging the relative position relationship between the side part of the vehicle body and the lane line.

[0118] Confirming the first region of interest through the first vehicle feature and identifying the lane feature and the position information of the second vehicle feature in the first region of interest can not only reduce the computing resources consumed in the recognition process, but also further improve the recognition efficiency and accuracy of the line of sight image.

[0119] In some embodiments, the image of the first region of interest may also be scaled in this step to improve recognition efficiency and accuracy.

[0120] 203. Determine whether the relative position relationship between the vehicle feature and the lane feature in the sight line image meets the preset reminder condition.

[0121] In one embodiment, the specific judgment process may include: judging whether the relative position relationship between the lane feature and the first vehicle feature meets the preset reminder condition; or judging whether the relative position relationship between the lane feature and the second vehicle feature meets the preset reminder condition.

[0122] It can be seen that the corresponding preset reminder conditions are set according to different project information, and then the position of the vehicle relative to the lane line at this time is determined by judging the relative position relationship between the lane feature and the first vehicle feature, or the relative position relationship between the lane feature and the second vehicle feature, so as to combine the requirements of different projects and more accurately judge the subsequent operations to be performed.

[0123] In some implementation scenarios, the first area of ​​interest is a rearview mirror, and the first area of ​​interest is a mirror area of ​​the rearview mirror. The position information of the vehicle features and lane features in the recognition line of sight image includes:

[0124] The mirror area is enlarged; the vehicle features and lane features in the enlarged image are identified; the pixel width of the lane features and the pixel width between the vehicle features and the lane features are obtained; and the relative distance and relative angle between the current lane features and the vehicle features are estimated based on the lane features, the pixel width between the vehicle features and the lane features, and the actual width of the lane.

[0125] 204. Obtain corresponding operation reminder information based on the recognition result.

[0126] Combination Figure 4 , the figure shows a schematic diagram of an application scenario of a driving training reminder method based on a wearable device provided in an embodiment of the present application.

[0127] In specific applications, such as Figure 4 As shown, for example, in the process of reversing into a garage, the position information of the rearview mirror 22 outside the window door 21 of the passenger side of the vehicle is first detected by the Mask R-CNN algorithm, and then the mirror area 23 of interest is cut out as the first region of interest (ROI area), and the vehicle feature and lane feature detection are performed on the area. The position of the vehicle feature 24 in the rearview mirror 22 and the position of the lane line 25 can be detected by Mask R-CNN. For the detected yellow lane line, the number of pixels occupied by the lane line width is counted in the image. Generally, the width of the yellow lane line is 10 cm. Based on the counted pixel width and the actual width of the lane line (10 cm), the angle (whether parallel) and distance between the current vehicle body and the lane line are estimated. Finally, according to the angle and distance, a voice broadcast is performed to prompt the trainee to perform operations such as steering wheel return.

[0128] Example 3

[0129] See also Figure 5 ,The figure shows the implementation process of obtaining the location information of the target lane features.

[0130] In one embodiment, if Figure 5 As shown, the preset features include at least vehicle features and lane features. The preset features in the sight line image are identified according to the project information, including:

[0131] 301. Obtaining position information of a first lane feature from the sight line image based on digital image detection;

[0132] 302. Obtain the location information of the second lane feature based on the Mask R-CNN deep learning algorithm;

[0133] 303. Obtain the position information corresponding to the target lane feature based on the first lane feature and the second lane feature.

[0134] Combine Figure 4 As shown, in this embodiment, the driving training reminder method is applied to a wearable device, and the wearable device is used to obtain a sight image of the user's sight direction.

[0135] Among them, the driving training item can be a reverse parking item. The user needs to perform the reverse parking operation by observing the rearview mirror 42. When the user's sight is directed outside the window door 21 in the co-pilot direction, the rearview mirror can be observed. At this time, the camera on the wearable device obtains a sight image of the user's sight direction.

[0136] The processing unit of the wearable device can first obtain the current item information, and the item information can include preset reminder conditions of vehicle features and lane features, and identify the sight image according to the item information. The preset reminder condition can identify the image of the rearview mirror 22 as the first vehicle feature, and the Mask R-CNN algorithm is used in the identification process.

[0137] In order to improve the accuracy of the lane lines 25 and other related lines segmented from the image, please refer to Figure 2 , first, detect the position of the rearview mirror 22 (the first vehicle feature) of the own driving vehicle through the Mask R-CNN network, and then intercept the mirror area 23 of the rearview mirror 22 in the image as the ROI (the first region of interest 23) region 23, where the ROI region 23 includes vehicle features 24 and lane lines 25. Take the image of the ROI region 23 as the target image, and further analyze the position of the lane line 25 in the target image with the help of traditional digital image-related knowledge. The specific operation is as follows: First, perform color space conversion on the obtained target image of the ROI region 23, convert the image from the RGB color space to the HSV color space, and then extract the H channel (the H color space represents the hue) of the image after converting the color space, and perform binary processing on this channel. Since the lane lines 25 on the training ground are generally yellow, two thresholds T1 and T2 (T1 < T2) are set, that is, the pixels of the H channel image between the T1 and T2 thresholds are initially determined to be the positions where the lane lines 25 are located, and other pixels are background pixel points. Further filter out non-linear yellow pixel points from the binary image through Hough transformation. Finally, generate a mask of the lane line 25 detected based on digital image processing.

[0138] In one embodiment, in order to improve the confidence of the lane line recognition result, the obtaining the position information corresponding to the target lane feature based on the first lane feature and the second lane feature includes:

[0139] Get the first lane area, the second lane area and the third lane area;

[0140] Performing similarity analysis on the first lane area, the second lane area, and the third lane area to obtain position information corresponding to the target lane feature;

[0141] Among them, the first lane area is the common area of ​​the first lane feature and the second lane feature; the second lane area is the lane area corresponding to the first lane feature and the second lane feature; the third lane area is the lane area corresponding to the lane feature detected in the historical sight line image. That is, the lane line obtained by Mask R-CNN segmentation and the lane line detected based on the digital image are logically ANDed to obtain the common lane line area obtained by the two different schemes, and then similarity analysis is performed based on the position of the commonly detected lane line, the position of the lane line detected by the two algorithms separately, and the position of the lane line detected in the historical sight line image, and the lane line position with higher similarity is retained, and finally the position information of the target lane feature is generated.

[0142] Through the above processing method, the characteristics and differences of the lane line positions detected in the two different detection methods and historical sight images can be used to correct the recognition results of the target lane features, thereby improving the confidence of the target lane feature recognition results.

[0143] Example 4

[0144] See also Figure 6 , the figure shows the structure of the vehicle-mounted system provided in an embodiment of the present application.

[0145] like Figure 6 As shown, the wearable device 400 includes a processor 401 and a memory 402, and the processor 401 and the memory 402 are electrically connected.

[0146] The wearable device 400 may include an image acquisition module 403 for providing to the user to wear. The wearable device can be worn on the head to obtain an image in the direction of the user's line of sight through the image acquisition module. In one embodiment, the image acquisition module 403 may be a camera, which may use an image sensor such as CCD, CMOS, etc. to sense and image the external light, and perform image processing through the image acquisition module 403 or the processing element in the wearable device 400 to obtain a line of sight image in the direction of the user's line of sight.

[0147] In order to obtain an image as close as possible to the user's line of sight, in one embodiment, the image acquisition module is a camera, and the camera is set in a position close to the human eye in the camera device. The camera is set near the user's eyes, such as the forehead, ear, etc. The wearable device 400 can be implemented in other wearable hardware forms such as a helmet, glasses, headphones, or fixed on the head. Furthermore, the camera device is smart glasses, and the camera is set between the frames on both sides of the smart glasses. Smart glasses are not only easy to use, but also can fit the user's eye position.

[0148] Through the wearable device 400, a line of sight image that is as close as possible to the user's line of sight can be obtained, thereby improving the accuracy and reliability of the operation reminder information.

[0149] The processor 401 is the control center of the wearable device 400. It uses various interfaces and lines to connect the various parts of the entire wearable device 400, and executes or executes the software programs and / or modules stored in the memory 402, and calls the data stored in the memory 402 to perform various functions and / or process data of the electronic wearable device 400. The processor 401 can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 401 can include only a central processing unit 401 (CPU). In the construction completion method of this application, the CPU can be a single computing core or multiple computing cores.

[0150] Among them, the memory 402 can be used to store the execution instructions of the processor 401, and the memory 402 can be implemented by any type of volatile or non-volatile storage wearable device 400 or a combination thereof, such as static random access memory 402 (SRAM), electrically erasable programmable read-only memory 402 (EEPROM), erasable programmable read-only memory 402 (EPROM), programmable read-only memory 402 (PROM), read-only memory 402 (ROM), magnetic memory 402, flash memory 402, disk or optical disk.

[0151] The memory 402 stores a computer program, and the processor 401 calls the computer program stored in the memory 402 to perform the following steps:

[0152] Acquire the sight line image through the image acquisition module; acquire the current project information, which is related to the current driving training project; identify the preset features in the sight line image according to the project information; acquire the corresponding operation reminder information based on the recognition result, and perform the display of the operation reminder information.

[0153] The image acquisition device 403 is electrically connected to the processor 401. In one embodiment, the number of cameras of the image acquisition device 403 may be 1, 2, or more than 2 to meet different image acquisition requirements.

[0154] The wearable device 400 may also include a reminder device, which may include a display, a speaker or other sound and light equipment.

[0155] In one embodiment, the wearable device 400 further includes an image output module, and the image output module is connected to the processor 401;

[0156] The processor 401 is further configured to execute:

[0157] Acquire driving operation information of the current vehicle; display the driving operation information in the sight line image and perform superposition processing; output the sight line image after superposition processing to the image output module, so as to perform image output through the image output module.

[0158] The driving operation information may include the vehicle speed, braking conditions, gear position, and number of steering wheel turns, so that the user's current driving operation can be reflected through the driving operation information. The driving operation information is displayed in the sight image and output, and students and coaches can obtain video data from the first-person perspective, which can facilitate students and coaches to review the entire process of driving training through the video data, and facilitate students to learn and summarize by themselves after driving practice, which is conducive to improving students' learning efficiency and learning effect.

[0159] In one embodiment, the processor 401 is further configured to execute:

[0160] Determine a corresponding preset reminder condition based on the project information, wherein the preset reminder condition includes a preset relative position relationship between the vehicle feature and the lane feature; identify the position information of the vehicle feature and the lane feature in the line of sight image; and determine whether the relative position relationship between the vehicle feature and the lane feature in the line of sight image meets the preset reminder condition.

[0161] In one embodiment, the processor 401 is further configured to execute:

[0162] A first vehicle feature is identified in the line of sight image; a first region of interest relative to the location of the first vehicle feature is determined; the position information of a lane feature is identified in the first region of interest, or the position information of a lane feature and a second vehicle feature is identified; it is determined whether the relative position relationship between the lane feature and the first vehicle feature meets a preset reminder condition; or it is determined whether the relative position relationship between the lane feature and the second vehicle feature meets a preset reminder condition.

[0163] The mirror area is enlarged; the vehicle feature and the lane feature in the enlarged image are identified; the pixel width of the lane feature and the pixel width between the vehicle feature and the lane feature are obtained; and the relative distance and relative angle between the current lane feature and the vehicle feature are estimated based on the lane feature, the pixel width between the vehicle feature and the lane feature, and the actual width of the lane.

[0164] In one embodiment, the processor 401 is further configured to execute:

[0165] The Mask R-CNN deep learning algorithm is used to identify the preset features in the line of sight image.

[0166] In one embodiment, the processor 401 is further configured to execute:

[0167] Based on digital image detection, the position information of the first lane feature is obtained from the line of sight image; based on the MaskR-CNN deep learning algorithm, the position information of the second lane feature is obtained; and based on the first lane feature and the second lane feature, the position information corresponding to the target lane feature is obtained.

[0168] In one embodiment, the processor 401 is further configured to execute:

[0169] A first vehicle feature is identified in the sight line image; a first region of interest relative to a location of the first vehicle feature is determined; a target image of the first region of interest is converted in color space so that the target image is converted from an RGB color space to an HSV color space; and the target image after the color space conversion is processed to obtain a lane feature mask of the target image.

[0170] In one embodiment, the processor 401 is further configured to execute:

[0171] Acquire the first lane area, the second lane area and the third lane area; perform similarity analysis on the first lane area, the second lane area and the third lane area to obtain position information corresponding to the target lane feature; wherein the first lane area is the common area of ​​the first lane feature and the second lane feature; the second lane area is the lane area corresponding to the first lane feature and the second lane feature respectively; the third lane area is the lane area corresponding to the lane feature detected in the historical sight image.

[0172] In the embodiment of the present application, the wearable device and the driving training reminder method based on the wearable device in the above embodiment belong to the same concept. Any method step provided in the embodiment of the driving training reminder method based on the wearable device can be run on the wearable device. The specific implementation process is detailed in the embodiment of the driving training reminder method based on the wearable device, and any combination can be used to form an optional embodiment of the present application, which will not be repeated here.

[0173] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0174] The implementation modes of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present application.

Claims

1. A driving training reminder method based on a wearable device, characterized in that: The method is applied to a wearable device provided to a user for wearing; The wearable device includes an image acquisition module, and the image acquisition module is oriented toward the user's line of sight; The method comprises: Acquire a sight line image through the image acquisition module; Acquiring current project information, where the project information is related to the currently ongoing driving training project; identifying preset features in the sight line image according to the project information; Acquire corresponding operation reminder information based on the recognition result, and perform display of the operation reminder information; The preset features include at least vehicle features and lane features; and identifying the preset features in the sight line image according to the project information includes: Determine a corresponding preset reminder condition according to the project information, wherein the preset reminder condition includes a preset relative position relationship between the vehicle feature and the lane feature; Identifying the position information of the vehicle feature and the lane feature in the sight line image, including identifying a first vehicle feature in the sight line image; determining a first region of interest relative to the location of the first vehicle feature; and identifying the position information of the lane feature in the first region of interest; Determining whether the relative position relationship between the vehicle feature in the sight line image and the lane feature meets the preset reminder condition includes determining whether the relative position relationship between the lane feature and the first vehicle feature meets the preset reminder condition.

2. The driving training reminder method based on a wearable device as claimed in claim 1, characterized in that: Identifying the position information of the vehicle features and the lane features in the sight line image also includes: Identifying the position information of the lane feature and the second vehicle feature in the first region of interest; The determining whether the relative position relationship between the vehicle feature in the sight line image and the lane feature meets the preset reminder condition also includes: It is determined whether a relative position relationship between the lane feature and the second vehicle feature meets a preset reminder condition.

3. The driving training reminder method based on a wearable device as claimed in claim 2, characterized in that: The first vehicle feature is a rear-view mirror, and the first region of interest is a mirror surface area of ​​the rear-view mirror; The identifying the position information of the vehicle features and the lane features in the sight line image includes: Enlarging the mirror area; Identify vehicle features and lane features in the magnified image; Obtaining the pixel width of the lane feature and the pixel width between the vehicle feature and the lane feature; The relative distance and relative angle between the current lane feature and the vehicle feature are estimated based on the lane feature, the pixel width between the vehicle feature and the lane feature, and the actual width of the lane.

4. The driving training reminder method based on a wearable device according to any one of claims 1 to 3, characterized in that: The identifying a preset feature in the sight line image according to the project information includes: The Mask R-CNN deep learning algorithm is used to identify the preset features in the line of sight image.

5. The driving training reminder method based on a wearable device as claimed in claim 4, characterized in that: The preset features include at least vehicle features and lane features; The identifying a preset feature in the sight line image according to the project information includes: obtaining position information of a first lane feature from the sight line image based on digital image detection; The location information of the second lane feature is obtained based on the Mask R-CNN deep learning algorithm; The position information corresponding to the target lane feature is obtained according to the first lane feature and the second lane feature.

6. The driving training reminder method based on a wearable device as claimed in claim 5, characterized in that: The obtaining the position information of the first lane feature from the sight line image based on digital image detection includes: identifying a first vehicle feature in the line of sight image; determining a first region of interest relative to a location of the first vehicle feature; Performing color space conversion on the target image of the first region of interest so that the target image is converted from the RGB color space to the HSV color space; The target image after the color space conversion is processed to obtain the lane feature mask of the target image.

7. The driving training reminder method based on a wearable device as claimed in claim 5, characterized in that: The obtaining, according to the first lane feature and the second lane feature, position information corresponding to the target lane feature includes: Get the first lane area, the second lane area and the third lane area; Performing similarity analysis on the first lane area, the second lane area, and the third lane area to obtain position information corresponding to the target lane feature; Among them, the first lane area is the common area of ​​the first lane feature and the second lane feature; the second lane area is the lane area corresponding to the first lane feature and the second lane feature respectively; the third lane area is the lane area corresponding to the lane feature detected in the historical sight image.

8. A wearable device, characterized in that: The wearable device includes an image acquisition module, and the image acquisition module is oriented toward the user's line of sight; The wearable device further includes a processor and a memory, wherein the processor is electrically connected to the memory; A computer program is stored in the memory, and the processor executes the driving training reminder method based on a wearable device as described in any one of claims 1 to 7 by calling the computer program stored in the memory.

9. The wearable device according to claim 8, wherein: The wearable device is a camera device worn on a human head; The image acquisition module is a camera, and the camera is arranged at a position close to the human eye in the camera device.

10. The wearable device according to claim 9, wherein: The camera device is a pair of smart glasses, and the camera is arranged between the frames on both sides of the smart glasses.

11. The wearable device according to claim 8, wherein: The wearable device further includes an image output module, and the image output module is connected to the processor; The processor is further configured to execute: Obtain driving operation information of the current vehicle; Displaying the driving operation information in the sight line image and performing superimposition processing; The sight line image after the superposition processing is output to the image output module so as to output the image through the image output module.

Images