Riding prompting method and device, vehicle, storage medium and computer program product

Abstract

The embodiment of the invention relates to a riding prompting method and device, a vehicle, a storage medium and a computer program product. The method comprises the following steps: acquiring multiple frames of target images arranged according to a time sequence; utilizing a preset rule engine and an artificial intelligence engine to carry out traffic signal identification processing on each frame of the target image to obtain a candidate area in each frame of the target image; determining target change information of a traffic signal according to the candidate area in each frame of the target image; and outputting riding prompt information according to the target change information of the traffic signal. By adopting the method, the riding safety of the user can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of cycling prompts, and in particular, to a cycling prompt method, device, vehicle, storage medium, and computer program product. Background Art

[0002] With the development of technology, shared two-wheel vehicles have flourished and become an indispensable means of transportation in people's lives. Currently, how to improve the cycling safety of users is a very important research direction. Summary of the Invention

[0003] Embodiments of the present disclosure provide a cycling prompt method, device, vehicle, storage medium, and computer program product, which can timely remind users to change their cycling methods according to the changes in traffic signals, thereby improving the cycling safety of users.

[0004] In a first aspect, embodiments of the present disclosure provide a cycling prompt method, which includes:

[0005] Obtain multiple target images arranged in chronological order;

[0006] Use a pre-set rule engine and an artificial intelligence engine to respectively perform traffic signal recognition processing on each frame of the target image to obtain candidate regions in each frame of the target image;

[0007] Determine the target change information of the traffic signal according to the candidate regions in each frame of the target image;

[0008] Output cycling prompt information according to the target change information of the traffic signal.

[0009] In a second aspect, embodiments of the present disclosure provide a cycling prompt device, which includes:

[0010] An image acquisition module, configured to obtain multiple target images arranged in chronological order;

[0011] An image recognition module, configured to use a pre-set rule engine and an artificial intelligence engine to respectively perform traffic signal recognition processing on each frame of the target image to obtain candidate regions in each frame of the target image;

[0012] A signal change determination module, configured to determine the target change information of the traffic signal according to the candidate regions in each frame of the target image;

[0013] A prompt module, configured to output cycling prompt information according to the target change information of the traffic signal.

[0014] In a third aspect, an embodiment of the present disclosure provides a vehicle, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method described in the first aspect above is implemented.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0017] The riding prompt method, device, vehicle, storage medium, and computer program product provided by the embodiments of the present disclosure acquire multiple target images arranged in chronological order; respectively perform traffic signal recognition processing on each frame of the target images by using a pre-set rule engine and an artificial intelligence engine to obtain candidate regions in each frame of the target images; determine target change information of the traffic signal according to the candidate regions in each frame of the target images; and output riding prompt information according to the target change information of the traffic signal. By collecting images in real time and using the rule engine and the artificial intelligence engine for image recognition to determine the change situation of the traffic signal, the embodiments of the present disclosure can timely remind the user to change the riding mode according to the change situation of the traffic signal, thereby improving the riding safety of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an application environment diagram of the riding prompt method in an embodiment;

[0019] Figure 2 It is a flowchart of the riding prompt method in an embodiment;

[0020] Figure 3 It is a flowchart of the step of determining candidate regions in an embodiment;

[0021] Figure 4 It is a flowchart of the step of determining candidate regions in an embodiment;

[0022] Figure 5 It is a flowchart of the step of determining target change information in an embodiment;

[0023] Figure 6 It is a flowchart of the step of outputting riding prompt information in an embodiment;

[0024] Figure 7 It is a flowchart of the step of outputting riding prompt information in an embodiment;

[0025] Figure 8 Schematic flowchart of steps for utilizing feedback information in an embodiment

[0026] Figure 9 Block diagram of a riding prompt device in an embodiment

[0027] Figure 10 Internal structure diagram of a vehicle in an embodiment Specific implementation manners

[0028] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present disclosure, and are not used to limit the embodiments of the present disclosure.

[0029] First, before specifically introducing the technical solutions of the embodiments of the present disclosure, the technical background or the technical evolution context on which the embodiments of the present disclosure are based will be introduced. With the development of technology, shared two-wheeled vehicles have flourished and become an indispensable means of transportation in people's lives. When users are riding two-wheeled vehicles, they occasionally run red lights without paying attention to traffic lights, or fail to ride in time after the green light. These behaviors all affect the safety of users. Therefore, how to improve the riding safety of users is a very important research direction. It should be noted that the applicant has put in a lot of creative labor from determining that users' failure to pay attention to traffic lights affects their riding safety and the technical solutions introduced in the following embodiments.

[0030] Next, the technical solutions related to the embodiments of the present disclosure will be introduced in combination with the scenarios to which the embodiments of the present disclosure are applied.

[0031] The riding prompt method provided by the embodiments of the present disclosure can be applied to, for example Figure 1In the application environment shown. The application environment includes a vehicle 102 and a server 104. Among them, an image acquisition device, a communication device, a prompting device, and a processor are provided on the vehicle 102. The vehicle 102 acquires an image in front of the vehicle through the image acquisition device, processes the image through the processor, and controls the prompting device to output a cycling prompt message. A rule engine and an artificial intelligence engine are set in the processor, and a feature extraction algorithm, a decision algorithm, a probability model, etc. can also be set. The vehicle 102 can communicate with the server 104 through the communication device, obtain the rule engine, the artificial intelligence engine, the feature extraction algorithm, the decision algorithm, the probability model, and the update information of each model and algorithm from the server 104, and can also upload cycling-related information, such as feedback information, to the server 104 for the server 104 to store or optimize the model. The above image acquisition device can include, but is not limited to, various low-power high-resolution cameras, cameras, and radars, etc. The above communication device can include, but is not limited to, a mobile communication module and a short-range communication module, etc. The above prompting device can include a display screen, a speaker, and an alarm light, etc.

[0032] In one embodiment, as Figure 2 shown, a cycling prompt method is provided. Taking the vehicle in Figure 1 as an example for description, the method includes the following steps:

[0033] Step 201, obtain multiple target images arranged in chronological order.

[0034] During the user's cycling process, the image acquisition device of the vehicle can collect images according to a preset period, and then transmit a preset number of collected images to the processor. For example, the image acquisition device collects one frame of image every 5 seconds. After collecting 5 frames of images, the 5 frames of images are transmitted to the processor. It should be noted that the preset period and the preset number can be set according to the actual situation and are not limited to the above description.

[0035] In some embodiments, the image acquisition device can use light adaptation technology to collect clear images in different environments.

[0036] The processor obtains the initial images collected by the image acquisition device, and performs image preprocessing on the initial images to obtain target images. Among them, the image preprocessing can include at least one of dynamic range compression and noise suppression.

[0037] The above dynamic range compression is a technique in image processing, which is used to adjust the contrast of an image so that details in an image captured under extreme lighting conditions become clearer. In the embodiments of the present disclosure, this technique can help reduce the influence of overexposed or underexposed areas, thereby making traffic signals easier to identify. To implement this technique, the following methods can be adopted: 1) Local contrast enhancement, that is, by increasing the local contrast in the image, making the traffic signal more prominent in the background; 2) HDR (High Dynamic Range Imaging): Combining images with different exposure levels into one image to obtain the maximum dynamic range and ensure that the traffic signal is clearly visible under any conditions; 3) Exposure fusion: Synthesizing multiple images with different exposure levels into a frame with balanced exposure.

[0038] The above noise suppression is mainly used to remove random variations generated during the image capture process, which will interfere with the recognition of traffic signals. Noise usually comes from low light, high ISO settings, or thermal noise of the sensor. The methods of noise suppression can include: 1) Temporal filtering: Using the method of multi-frame averaging to reduce random noise by synthesizing multiple continuously captured frames; 2) Spatial filtering: Using noise reduction algorithms such as median filtering and Gaussian filtering can effectively suppress random noise points in the image; 3) Frequency domain filtering: Adjusting the frequency components in the Fourier transform of the image to eliminate the noise frequency while retaining the image details.

[0039] Step 202, respectively perform traffic signal recognition processing on each frame of target image by using a pre-set rule engine and an artificial intelligence engine to obtain candidate regions in each frame of target image.

[0040] A rule engine and an artificial intelligence engine are set in the processor of the vehicle. The rule engine can perform traffic signal recognition processing on each frame of target image based on pre-set rules, and the artificial intelligence engine can perform traffic signal recognition processing based on a deep learning model.

[0041] In practical applications, the recognition results of the rule engine and the artificial intelligence engine can be comprehensively considered. For example, perform weighted summation processing on the recognition results of the rule engine and the artificial intelligence engine to obtain a comprehensive recognition result, and then extract candidate regions containing traffic signals from the target image according to the comprehensive recognition result.

[0042] The above rule engine performs recognition processing based on pre-set rules, which can include: first, according to the pre-set segmentation rules, segment the target recognition region from the target image; then perform color recognition processing on the target recognition region according to the pre-set color recognition rules, and perform shape recognition on the target recognition region according to the pre-set shape recognition rules to obtain the recognition result.

[0043] Among them, the above-mentioned preset segmentation rules can be determined according to the expected position and size of the traffic signal in the image; the above-mentioned preset color recognition rules can be determined according to the color of the traffic signal; the above-mentioned preset shape recognition rules can be determined according to the shape of the traffic signal.

[0044] The above color recognition processing may include: performing a color space transformation on the image, such as converting from RGB to HSV or HSL; and then marking the pixels in the target recognition area according to the preset color threshold. The above color space transformation processing can improve the separation and recognition efficiency of colors.

[0045] The above shape recognition processing may include: using a geometric shape detection algorithm to perform recognition processing on the target recognition area. For example, using the Hough transform algorithm to recognize circular traffic signals and rectangular traffic signs; or, combining morphological operations, such as erosion processing and dilation processing, to improve the detection of shape features.

[0046] It should be noted that the color recognition processing and the shape recognition processing are not limited to the above-described methods. In practical applications, other methods can also be used to achieve them.

[0047] The deep learning model adopted by the above artificial intelligence engine can be a Convolutional Neural Network (CNN) model, which can recognize various traffic signals under different lighting and weather conditions; it can also be a transfer learning model, such as ImageNet, which can improve the recognition accuracy and generalization ability. The deep learning model can be trained by a server. During the training process, the server can adopt various data augmentation techniques, such as rotation, scaling, shearing, color jitter, etc.; after the training is completed, the server can also continuously learn and optimize, so as to continuously improve the recognition adaptability and accuracy.

[0048] Step 203, determine the target change information of the traffic signal according to the candidate regions in each frame of the target image.

[0049] Among them, the target change information characterizes the change situation of the traffic signal. For example, changing from green light to yellow light, from yellow light to red light, from red light to green light, etc. Further, according to the shape of the traffic signal, the target change information can also include the change situation of the straight traffic signal, the change situation of the left-turn traffic signal, etc.

[0050] The multi-frame target images are arranged in chronological order. Therefore, by analyzing and processing the candidate regions of the multi-frame target images, the change situation of the traffic signal can be determined to obtain the target change information. For example, if it is determined that the candidate regions in the first and second frames of the target images are red lights, and the candidate regions in the third, fourth, and fifth frames of the target images are green lights, then the target change information can be determined as changing from red light to green light.

[0051] Step 204: Output a cycling prompt message according to the target change information of the traffic signal.

[0052] Among them, the cycling prompt message is used to prompt the user to change the cycling mode. For example, the cycling prompt message includes "Please start cycling", "Please stop cycling", "Please reduce the cycling speed", etc.

[0053] After determining the target change information of the traffic signal, a cycling prompt message can be output according to the target change information. For example, if the target change information is from red light to green light, the vehicle can output a cycling prompt message of "Please start cycling".

[0054] There are various ways for the vehicle to output the cycling prompt message. For example, the cycling prompt message is displayed on the display screen, or the corresponding identifier of the cycling prompt message; the cycling prompt message is played through the speaker; the corresponding cycling prompt message is illuminated by the warning light.

[0055] It should be noted that the cycling prompt message and the information output method are not limited to the above description. In actual applications, other cycling prompt messages and other information output methods can also be adopted.

[0056] In the above embodiment, multiple frames of target images arranged in chronological order are acquired; the rule engine and the artificial intelligence engine set in advance are respectively used to perform traffic signal recognition processing on each frame of target image to obtain candidate regions in each frame of target image; the target change information of the traffic signal is determined according to the candidate regions in each frame of target image; a cycling prompt message is output according to the target change information of the traffic signal. In the embodiment of the present disclosure, images are collected in real time, and the rule engine and the artificial intelligence engine are used for image recognition to determine the change situation of the traffic signal. In this way, the user can be reminded to change the cycling mode in time according to the change situation of the traffic signal, thereby improving the cycling safety of the user.

[0057] In one embodiment, as Figure 3 shown, the process of using the rule engine and the artificial intelligence engine set in advance to perform traffic signal recognition processing on each frame of target image to obtain candidate regions in each frame of target image may include the following steps:

[0058] Step 301: Input each frame of target image into the rule engine in sequence for traffic signal recognition processing, and input each frame of target image into the artificial intelligence engine in sequence for traffic signal recognition processing.

[0059] In practical applications, since the reminder during cycling has relatively high requirements for real-time performance, the rule engine and the artificial intelligence engine can perform parallel recognition. That is, each frame of target image is sequentially input into the rule engine for traffic signal recognition processing according to the time sequence, and at the same time, each frame of target image is also sequentially input into the artificial intelligence engine for traffic signal recognition processing. The specific recognition process can refer to the description of the above embodiments, and the embodiments of the present disclosure will not be elaborated herein.

[0060] Step 302: Determine the candidate regions in each frame of target image according to the recognition results output first by the rule engine and the artificial intelligence engine.

[0061] Among them, the recognition results may include the position and confidence of the traffic signal light, and may also include the category of the traffic signal light.

[0062] There are usually differences in the recognition efficiency between the rule engine and the artificial intelligence engine. Therefore, one of the engines often outputs the recognition result first. Considering the real-time requirements, the candidate regions in each frame of target image can be determined according to the recognition result output first.

[0063] In the above embodiments, each frame of target image is sequentially input into the rule engine for traffic signal recognition processing, and each frame of target image is sequentially input into the artificial intelligence engine for traffic signal recognition processing; the candidate regions in each frame of target image are determined according to the recognition results output first by the rule engine and the artificial intelligence engine. The embodiments of the present disclosure utilize the parallel recognition of the rule engine and the artificial intelligence engine, which can improve the recognition efficiency and thus meet the real-time requirements of the reminder.

[0064] In one embodiment, the image acquisition device of the vehicle may be blocked by the pedestrians and vehicles in front, resulting in the image collected by the image acquisition device not containing the traffic signal light. In this case, when the rule engine and the artificial intelligence engine perform traffic signal recognition processing on the target image, the problem of not being able to recognize the traffic signal light will occur. And if the traffic signal light cannot be recognized, it is difficult to remind the user in time.

[0065] In view of the above problems, in one scenario, as Figure 4 shown, the process of determining the candidate regions in each frame of target image according to the recognition results output first by the rule engine and the artificial intelligence engine may include the following steps:

[0066] Step 3021: When the rule engine and the artificial intelligence engine do not recognize the traffic signal light in multiple frames of target images, use the rule engine and the artificial intelligence engine to recognize the taillights of the vehicle in front in each frame of target image.

[0067] If the rule engine and the artificial intelligence engine do not recognize a traffic signal in the multi-frame target images, the rule engine and the artificial intelligence engine are used to recognize the taillights of the vehicle in front in each frame of the target images. The specific recognition method can refer to the above-mentioned traffic signal recognition method, and the embodiments of the present disclosure will not elaborate herein.

[0068] Step 3022: Determine the candidate regions in each frame of the target images according to the taillight recognition result output first by the rule engine and the artificial intelligence engine.

[0069] There are differences in the recognition speeds of the rule engine and the artificial intelligence engine. One of the rule engine and the artificial intelligence engine outputs the taillight recognition result first, and the candidate regions containing the taillights of the vehicle in front are extracted from each frame of the target images according to the taillight recognition result.

[0070] Regarding the above problem, in another scenario, the process of determining the candidate regions in each frame of the target images according to the recognition result output first by the rule engine and the artificial intelligence engine may include: when the rule engine and the artificial intelligence engine do not recognize a traffic signal in the multi-frame target images, obtaining the target change information of the traffic signal from other vehicles and / or roadside intelligent devices.

[0071] If the rule engine and the artificial intelligence engine do not recognize a traffic signal in the multi-frame target images, a short-range communication connection can be established with other vehicles, and the target change information of the traffic signal can be obtained from other vehicles through the short-range communication connection. A communication can also be established with the roadside intelligent device to obtain the target change information of the traffic signal from the roadside intelligent device.

[0072] The above-mentioned vehicles are not limited to two-wheeled vehicles, automobiles, etc. The above-mentioned roadside intelligent devices may include, but are not limited to, traffic signals, communication base stations, etc.

[0073] It should be noted that in the case where the image acquisition device of the vehicle is blocked, the solution is not limited to the above description, and other methods can also be adopted.

[0074] In the above embodiments, when the rule engine and the artificial intelligence engine fail to recognize traffic lights in multiple frames of target images, the rule engine and the artificial intelligence engine are used to recognize the taillights of the vehicle in front in each frame of target image; according to the taillight recognition results output first by the rule engine and the artificial intelligence engine, candidate regions in each frame of target image are determined; alternatively, when the rule engine and the artificial intelligence engine fail to recognize traffic lights in multiple frames of target images, target change information of traffic signals is obtained from other vehicles and / or roadside intelligent devices. By recognizing the taillights of the vehicle in front to determine the change of traffic signals or obtaining the change of traffic signals from other vehicles and roadside intelligent devices, the embodiments of the present disclosure can solve the problem that it is difficult to remind users when the image acquisition device is blocked, and improve the reliability of reminder.

[0075] In one embodiment, as Figure 5 shown, the process of determining the target change information of traffic signals according to the candidate regions in each frame of target image may include the following steps:

[0076] Step 401: Extract features from the candidate regions in each frame of target image to obtain a plurality of feature vectors.

[0077] Among them, the feature vectors can characterize the features of traffic lights in the candidate regions. For example, the feature vectors characterize the color and / or shape of traffic lights, etc.

[0078] A feature extraction algorithm can be set in the processor of the vehicle. After determining the candidate regions in each frame of target image, the candidate regions can be cropped from each frame of target image, and the feature extraction algorithm can be used to perform feature extraction processing on each candidate region to obtain a plurality of feature vectors.

[0079] The feature extraction algorithm may include but is not limited to SIFT (Scale-invariant feature transform) and SURF (Speeded Up Robust Features), which can extract key points and descriptors from the candidate regions. Among them, the key points are specific points in the image, such as the corners, edges or spots of an object; the descriptors are mathematical descriptions of the regions around the key points, providing appearance information of the key points.

[0080] The above SIFT is an algorithm for detecting and describing key points in an image. SIFT features are invariant to rotation, scale scaling, brightness changes and partial perspective changes. The above SURF is an improved version of SIFT, aiming to speed up the feature extraction while maintaining similar recognition efficiency. It usually uses integral images to quickly calculate the descriptors of the regions around the interest points.

[0081] It should be noted that the feature extraction algorithm is not limited to the algorithms described above. In practical applications, other algorithms can also be adopted.

[0082] Step 402: Determine the target change information of the traffic signal according to the temporal relationship of multiple feature vectors.

[0083] Since multiple frames of target images are arranged in chronological order, there is also a temporal relationship between multiple feature vectors. According to this temporal relationship, the target change information of the traffic signal can be determined.

[0084] For example, if the feature vector represents the color of the traffic signal, and the temporal relationship of multiple feature vectors is that the colors are red, red, green, green, green from front to back, the target change information of the traffic signal can be determined as changing from red light to green light according to this temporal relationship. The feature vector can also include the shape of the traffic signal. According to the temporal relationship of color and shape, the target change information of the left-turn traffic signal, the target change information of the straight-ahead traffic signal, etc. can be determined.

[0085] In the above embodiments, feature extraction is performed on the candidate regions in each frame of the target image to obtain multiple feature vectors; according to the temporal relationship of the multiple feature vectors, the target change information of the traffic signal is determined. In the embodiments of the present disclosure, feature extraction is performed on the candidate regions, which can reduce the computational amount of feature extraction, thereby improving the speed of feature extraction, and further meeting the real-time requirement of the reminder.

[0086] In one embodiment, as Figure 6 shown, the process of outputting the cycling reminder information according to the target change information of the traffic signal may include the following steps:

[0087] Step 501: Obtain the cycling speed of the user and the target distance between the vehicle and the traffic signal.

[0088] The vehicle can also be provided with a positioning device, and the processor of the vehicle determines the cycling speed of the user and the target distance between the vehicle and the traffic signal according to the vehicle position collected by the positioning device.

[0089] It should be noted that the vehicle can also be provided with other devices, and the above cycling speed and target distance are determined according to the data collected by the other devices.

[0090] Step 502: Output the cycling reminder information according to the cycling speed, the target distance, and the target change information of the traffic signal.

[0091] Among them, the target change information of traffic signals includes the target change information of traffic lights or the taillights of the vehicle in front. A decision-making algorithm can be set in the vehicle's processor. After determining the riding speed, target distance, and target change information of traffic signals, the decision-making algorithm calculates the riding speed, target distance, and target change information of traffic signals to generate a riding instruction. Then, the vehicle's processor transmits the riding instruction to the prompting device, and the prompting device outputs riding prompt information according to the prompting instruction.

[0092] The above decision-making algorithm can include at least one of decision trees, support vector machines, and neural network models. Among them, a decision tree is a simple and intuitive classification method that can gradually subdivide the data set and make judgments based on a series of rules. Support Vector Machines (SVM) is a powerful classification algorithm that finds the best separating hyperplane in the data feature space. For non-linearly separable problems, SVM can be solved by mapping to a high-dimensional space through the kernel trick. A neural network model is a complex model with a multi-layer structure that can learn the deep feature representation of data through training.

[0093] It should be noted that the decision-making algorithm is not limited to the above description. In practical applications, other algorithms can also be used.

[0094] In the above embodiment, the riding speed of the user and the target distance between the vehicle and the traffic signal are obtained; according to the riding speed, target distance, and target change information of the traffic signal, riding prompt information is output. The embodiment of the present disclosure reminds the user based on the change situation of the traffic signal, which can effectively improve the riding safety of the user.

[0095] In some embodiments, as Figure 7 shown, the process of outputting riding prompt information according to the riding speed, target distance, and target change information of the traffic signal can include the following steps:

[0096] Step 5021, use the decision-making algorithm to determine the initial riding instruction and the confidence level corresponding to the initial riding instruction according to the riding speed, target distance, and target change information of the traffic signal.

[0097] Among them, the confidence level represents the probability that the instruction is correct.

[0098] Input the riding speed, target distance, and target change information of the traffic signal into decision-making algorithms such as decision trees and SVMs. The decision-making algorithm outputs the initial riding instruction and the confidence level corresponding to the initial riding instruction. For example, the decision-making algorithm outputs a start riding instruction, and the confidence level of this start riding instruction is 90%.

[0099] Step 5022, when the confidence level corresponding to the initial riding instruction is less than the preset confidence threshold, use the preset probability model to update the initial riding instruction, and output a riding prompt message according to the updated riding instruction.

[0100] After obtaining the confidence level of the initial riding instruction, compare the confidence level of the initial riding instruction with the preset confidence threshold. If the confidence level of the initial riding instruction is greater than or equal to the preset confidence threshold, it indicates that the probability of the initial riding instruction being correct is relatively high, and a riding prompt message is output according to the initial riding instruction. If the confidence level of the initial riding instruction is less than the preset confidence threshold, it indicates that the probability of the initial riding instruction being correct is relatively low. Then, a conservative strategy can be adopted, that is, maintaining the current riding state, or prompting the user to confirm manually. Or, use a probability model to evaluate the possibility of different signal states, and select the signal state with the highest probability to update the initial riding instruction to obtain the updated riding instruction. Then, output a riding prompt message according to the updated riding instruction.

[0101] The above probability model can adopt a Bayesian network, a random forest, etc., and can be set according to the actual situation in practical applications.

[0102] In the above embodiments, use a decision algorithm to determine the initial riding instruction and the confidence level corresponding to the initial riding instruction according to the riding speed, the target distance, and the target change information of the traffic signal; when the confidence level corresponding to the initial riding instruction is less than the preset confidence threshold, use the preset probability model to update the initial riding instruction, and output a riding prompt message according to the updated riding instruction. By judging the confidence level and the probability model in the embodiments of the present disclosure, the correctness of the riding instruction can be effectively improved, so as to better remind the user.

[0103] In one embodiment, as Figure 8 shown, the embodiments of the present disclosure may further include the following steps:

[0104] Step 601, obtain feedback information corresponding to the user's riding behavior.

[0105] Among them, the feedback information is used to characterize whether the user's riding behavior conforms to the riding prompt message.

[0106] After outputting the riding prompt message, determine the user's riding behavior and generate feedback information according to the user's riding behavior. For example, if the output riding prompt message is "Please start riding" and it is determined that the user starts riding, the generated feedback information is that the user's riding behavior conforms to the riding prompt message. Another example is that if the output riding prompt message is "Please stop riding" and it is determined that the user is still riding, the generated feedback information is that the user's riding behavior does not conform to the riding prompt message.

[0107] Step 602: Send feedback information to the server for the server to optimize the decision algorithm; or update the riding prompt information according to the feedback information.

[0108] The feedback information can be used in two stages. One stage is the training or optimization stage of the decision algorithm, and the other stage is the usage stage of the decision algorithm.

[0109] In the training or optimization stage of the decision algorithm, after using the decision algorithm to output riding prompt information and obtaining the feedback information, the vehicle sends the feedback information to the server; the server receives the feedback information. If the feedback information indicates that the user's riding behavior conforms to the riding prompt information, it means that the probability that the riding instruction generated by the decision algorithm is correct is relatively high, and this riding prompt information can be used as a positive sample for training the decision algorithm; if the feedback information indicates that the user's riding behavior does not conform to the riding prompt information, it means that the probability that the riding instruction generated by the decision algorithm is correct is relatively low, and this riding prompt information can be used as a negative sample for training the decision algorithm. The server optimizes the decision algorithm according to the positive samples and negative samples. Then, the server sends the algorithm update information obtained from the optimization process to the vehicle, and the vehicle can update the decision algorithm according to the algorithm update information.

[0110] It can be understood that the feedback information can not only be used to optimize the decision algorithm, but also be used to optimize the rule engine, artificial intelligence engine, feature extraction algorithm, etc.

[0111] In the usage stage of the decision algorithm, after using the decision algorithm to output riding prompt information and obtaining the feedback information, if the feedback information indicates that the user's riding behavior conforms to the riding prompt information, it means that the user rides according to the riding prompt information. If the feedback information indicates that the user's riding behavior does not conform to the riding prompt information, it means that the user does not ride according to the riding prompt information. In this case, the riding prompt information can be updated to increase the urgency and intensity of the prompt.

[0112] For example, the output riding prompt information is "Please stop riding", but the feedback information indicates that the user's riding behavior does not conform to the riding prompt information, then a new riding prompt information "Please stop riding immediately" can be output.

[0113] It should be noted that updating the riding prompt information can change the information expression of the riding prompt information, the information prompt frequency, the prompt volume of the speaker, the flashing frequency of the warning light, etc.

[0114] In the above embodiments, obtain the feedback information corresponding to the user's riding behavior; send the feedback information to the server for the server to optimize the decision algorithm; or update the riding prompt information according to the feedback information. The present disclosure can achieve the effect of improving the user's riding safety by obtaining user feedback to optimize the decision algorithm or increasing the urgency and intensity of the prompt.

[0115] In one embodiment, a cycling prompt method is provided. Taking the vehicle in Figure 1 as an example, the method includes the following steps:

[0116] Step 1: Obtain multiple target images arranged in chronological order,

[0117] Step 2: Input each frame of the target image into a rule engine for traffic signal recognition processing in sequence, and input each frame of the target image into an artificial intelligence engine for traffic signal recognition processing in sequence.

[0118] After the recognition processing, execute Step 3 or Step 4 or Step 8.

[0119] Step 3: In the case where the rule engine and the artificial intelligence engine recognize traffic lights in multiple frames of target images, determine the candidate regions in each frame of the target image according to the recognition result output first in the rule engine and the artificial intelligence engine.

[0120] After determining the candidate regions, execute Step 6.

[0121] Step 4: In the case where the rule engine and the artificial intelligence engine do not recognize traffic lights in multiple frames of target images, use the rule engine and the artificial intelligence engine to recognize the taillights of the vehicle in front in each frame of the target image.

[0122] Step 5: Determine the candidate regions in each frame of the target image according to the taillight recognition result output first in the rule engine and the artificial intelligence engine.

[0123] After determining the candidate regions, execute Step 6.

[0124] Step 6: Extract features from the candidate regions in each frame of the target image to obtain multiple feature vectors.

[0125] Step 7: Determine the target change information of the traffic signal according to the temporal relationship of the multiple feature vectors.

[0126] After determining the target change information, execute Step 9.

[0127] Step 8: In the case where the rule engine and the artificial intelligence engine do not recognize traffic lights in multiple frames of target images, obtain the target change information of the traffic signal from other vehicles and / or roadside intelligent devices.

[0128] After determining the target change information, execute Step 9.

[0129] Step 9: Obtain the cycling speed of the user and the target distance between the vehicle and the traffic signal.

[0130] Step 10: Using a decision-making algorithm, determine an initial riding instruction and the confidence level corresponding to the initial riding instruction based on the riding speed, the target distance, and the target change information of the traffic signal.

[0131] Step 11: In the case where the confidence level corresponding to the initial riding instruction is less than a preset confidence threshold, use a preset probability model to update the initial riding instruction, and output a riding prompt message according to the updated riding instruction.

[0132] Step 12: Obtain feedback information corresponding to the user's riding behavior;

[0133] Step 13: Send the feedback information to the server for the server to update the decision-making algorithm; or update the riding prompt message according to the feedback information.

[0134] In some embodiments, the vehicle can also generate a safety log to record relevant data during the riding process, and the data is crucial for accident investigation, system performance evaluation, and future safety improvement.

[0135] In some embodiments, the riding prompt message can also be in a user-defined manner, thereby enhancing the user's usage experience and satisfaction.

[0136] In some embodiments, an emergency handling strategy can also be set. When a potential danger or emergency situation is detected, emergency handling measures can be taken, such as communicating with an emergency service provider or activating other safety features, thereby further enhancing the user's riding safety.

[0137] It should be understood that although Figures 2 to 8 the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 2 to 8 at least a part of the steps in

[0138] In one embodiment, as Figure 9 shown, a riding prompt device is provided, including:

[0139] An image acquisition module 701, configured to acquire multiple frames of target images arranged in chronological order;

[0140] An image recognition module 702 is configured to perform traffic signal recognition processing on each frame of target image by using a preset rule engine and an artificial intelligence engine respectively, so as to obtain candidate regions in each frame of target image;

[0141] A signal change determination module 703 is configured to determine target change information of a traffic signal according to candidate regions in each frame of target image;

[0142] A prompt module 704 is configured to output a cycling prompt message according to the target change information of the traffic signal.

[0143] In some embodiments, the image recognition module 702 is specifically configured to sequentially input each frame of target image into the rule engine for traffic signal recognition processing, and sequentially input each frame of target image into the artificial intelligence engine for traffic signal recognition processing; and determine candidate regions in each frame of target image according to the recognition results output first by the rule engine and the artificial intelligence engine.

[0144] In some embodiments, the image recognition module 702 is specifically configured to, when the rule engine and the artificial intelligence engine do not recognize traffic signal lights in multiple frames of target images, use the rule engine and the artificial intelligence engine to recognize the taillights of the vehicle in front in each frame of target image; and determine candidate regions in each frame of target image according to the taillight recognition results output first by the rule engine and the artificial intelligence engine.

[0145] In some embodiments, the device further includes:

[0146] A signal change acquisition module is configured to, when the rule engine and the artificial intelligence engine do not recognize traffic signal lights in multiple frames of target images, obtain the target change information of the traffic signal from other vehicles and / or roadside intelligent devices.

[0147] In some embodiments, the signal change determination module 703 is specifically configured to extract features from candidate regions in each frame of target image to obtain a plurality of feature vectors; and determine the target change information of the traffic signal according to the temporal relationship of the plurality of feature vectors.

[0148] In some embodiments, the prompt module 704 is specifically configured to obtain the cycling speed of the user and the target distance between the vehicle and the traffic signal; and output a cycling prompt message according to the cycling speed, the target distance, and the target change information of the traffic signal.

[0149] In some embodiments, the prompting module 704 is specifically configured to use a decision algorithm to determine an initial riding instruction and the confidence level corresponding to the initial riding instruction according to the riding speed, the target distance, and the target change information of the traffic signal; in the case where the confidence level corresponding to the initial riding instruction is less than a preset confidence threshold, update the initial riding instruction by using a preset probability model, and output a riding prompt message according to the updated riding instruction.

[0150] In some embodiments, the device further includes:

[0151] a feedback information acquisition module, configured to acquire feedback information corresponding to the user's riding behavior;

[0152] an update module, configured to send the feedback information to the server for the server to optimize the decision algorithm; or update the riding prompt message according to the feedback information.

[0153] For the specific limitations of the riding prompt device, reference may be made to the limitations on the riding prompt method in the foregoing text, which will not be elaborated herein. Each module in the above-mentioned riding prompt device can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in or independent of the vehicle's processor in the form of hardware, or stored in the vehicle's memory in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0154] Figure 10 FIG. 16 is a block diagram of a vehicle 1300 shown according to an exemplary embodiment. The vehicle 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power supply component 1306, a multimedia component 1308, an audio component 1310, an input / output (I / O) interface 1312, a sensor component 1314, and a communication component 1316. Among them, a computer program or instruction is stored on the memory and runs on the processor.

[0155] The processing component 1302 generally controls the overall operation of the vehicle 1300, such as operations associated with display and data communication. The processing component 1302 may include one or more processors 1320 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 1302 may include one or more modules to facilitate the interaction between the processing component 1302 and other components. For example, the processing component 1302 may include a multimedia module to facilitate the interaction between the multimedia component 1308 and the processing component 1302.

[0156] The memory 1304 is configured to store various types of data to support the operation of the vehicle 1300. Examples of such data include instructions for any application or method operating on the vehicle 1300, pictures, videos, and the like. The memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0157] The power supply component 1306 provides power to various components of the vehicle 1300. The power supply component 1306 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the vehicle 1300.

[0158] The multimedia component 1308 includes a touch display screen that provides an output interface between the vehicle 1300 and the user. In some embodiments, the touch display screen can include a liquid crystal display (LCD) and a touch panel (TP). The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of a touch or swipe action but also detect the duration and pressure associated with the touch or swipe operation.

[0159] The audio component 1310 is configured to output and / or input audio signals. For example, the audio component 1310 includes a microphone (MIC) that is configured to receive external audio signals when the vehicle 1300 is in an operating mode, such as a recording mode and a voice recognition mode. The received audio signals can be further stored in the memory 1304 or transmitted via the communication component 1316. In some embodiments, the audio component 1310 further includes a speaker for outputting audio signals.

[0160] The sensor component 1314 includes one or more sensors for providing an assessment of the state of various aspects of the vehicle 1300. For example, the sensor component 1314 can also detect a change in the position of the vehicle 1300, the presence or absence of user contact with the vehicle 1300, the orientation of the vehicle 1300 or acceleration / deceleration, and a change in the temperature of the vehicle 1300. The sensor component 1314 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. In some embodiments, the sensor component 1314 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0161] The communication component 1316 is configured to facilitate communication between the vehicle 1300 and other devices in a wired or wireless manner. The vehicle 1300 can access a communication standard-based wireless network, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 1316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1316 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0162] In an exemplary embodiment, the vehicle 1300 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-mentioned riding prompt method.

[0163] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 1304 including instructions, and the above instructions can be executed by the processor 1320 of the vehicle 1300 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0164] In an exemplary embodiment, a computer program product is also provided. When the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in accordance with the process or function described in the embodiments of the present disclosure.

[0165] It should be noted that for the solutions described in this specification and the embodiments, if they involve personal information processing, they will be processed on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for performing a contract, etc.), and will only be processed within the specified or agreed scope. If a user refuses to process personal information other than the necessary information required for basic functions, it will not affect the user's use of basic functions.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0167] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0168] The above-described embodiments merely represent several implementation manners of the embodiments of the present disclosure. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present disclosure, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present disclosure. Therefore, the protection scope of the patent of the embodiments of the present disclosure should be subject to the appended claims.

Claims

1. A cycling prompt method, characterized in that, The method includes: Obtaining multiple target images arranged in chronological order; Using a pre-set rule engine and an artificial intelligence engine to perform traffic signal recognition processing on each frame of the target images respectively, to obtain candidate regions in each frame of the target images; Determining target change information of traffic signals according to the candidate regions in each frame of the target images; Outputting a cycling prompt message according to the target change information of the traffic signals.

2. The method according to claim 1, wherein The step of using a pre-set rule engine and an artificial intelligence engine to perform traffic signal recognition processing on each frame of the target images respectively, to obtain candidate regions in each frame of the target images includes: Sequentially inputting each frame of the target images into the rule engine for traffic signal recognition processing, and sequentially inputting each frame of the target images into the artificial intelligence engine for traffic signal recognition processing; Determining the candidate regions in each frame of the target images according to the recognition results output first by the rule engine and the artificial intelligence engine.

3. The method according to claim 2, wherein The step of determining the candidate regions in each frame of the target images according to the recognition results output first by the rule engine and the artificial intelligence engine includes: In the case where the rule engine and the artificial intelligence engine do not recognize traffic signal lights in multiple frames of the target images, using the rule engine and the artificial intelligence engine to recognize the taillights of the vehicle in front in each frame of the target images; Determining the candidate regions in each frame of the target images according to the recognition results of the taillights of the vehicle in front output first by the rule engine and the artificial intelligence engine.

4. The method according to claim 2, characterized in that The method further includes: In the case where the rule engine and the artificial intelligence engine do not recognize traffic signal lights in multiple frames of the target images, obtaining the target change information of the traffic signals from other vehicles and / or roadside intelligent devices.

5. The method according to claim 1, wherein The step of determining the target change information of the traffic signals according to the candidate regions in each frame of the target images includes: Performing feature extraction on the candidate regions in each frame of the target images to obtain a plurality of feature vectors; Determining the target change information of the traffic signals according to the temporal relationship of the plurality of feature vectors.

6. The method according to any one of claims 1-5, characterized in that, The step of outputting a cycling prompt message according to the target change information of the traffic signals includes: Obtaining the cycling speed of the user and the target distance between the vehicle and the traffic signals; Outputting a cycling prompt message according to the cycling speed, the target distance and the target change information of the traffic signals.

7. The method according to claim 6, wherein The step of outputting a cycling prompt message according to the cycling speed, the target distance and the target change information of the traffic signals includes: Using a decision algorithm to determine an initial cycling instruction and the confidence level corresponding to the initial cycling instruction according to the cycling speed, the target distance and the target change information of the traffic signals; In the case where the confidence level corresponding to the initial cycling instruction is less than a pre-set confidence threshold, using a pre-set probability model to perform update processing on the initial cycling instruction, and outputting the cycling prompt message according to the updated cycling instruction.

8. The method according to claim 7, wherein The method further includes: Obtaining feedback information corresponding to the cycling behavior of the user; Send the feedback information to the server for the server to optimize the decision algorithm; or update the cycling prompt information according to the feedback information.

9. A cycling prompt device, characterized in that, The device includes: An image acquisition module, configured to acquire multiple frames of target images arranged in chronological order; An image recognition module, configured to perform traffic signal recognition processing on each frame of the target images respectively by using a pre-set rule engine and an artificial intelligence engine to obtain candidate regions in each frame of the target images; A signal change determination module, configured to determine target change information of traffic signals according to the candidate regions in each frame of the target images; A prompt module, configured to output cycling prompt information according to the target change information of the traffic signals.

10. A vehicle, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1-8 are implemented.

Images