Safe driving reminding method and device and storage medium

Abstract

The invention relates to a safe driving reminding method and device and a storage medium. The safe driving reminding method comprises the steps that an image outside a vehicle is acquired, and a target object is recognized in the image; in response to the identified target object which is located in a preset region of interest, monitoring a moving track of the target object and a moving track of the vehicle; and performing safe driving reminding based on the moving track of the target object and the moving track of the vehicle. According to the invention, real-time safe driving reminding is realized through the terminal placed on the vehicle.

Description

Technical Field

[0001] The present disclosure relates to the technical field of safe driving, and in particular to a safe driving reminder method, device and storage medium. Background Art

[0002] With the development of science and technology and the improvement of people's living standards, more and more vehicles are pouring into the road, and safe driving can no longer be ignored.

[0003] In the related technology, by using map navigation software, safe driving reminders can be given while the vehicle is driving, but the reminders of map navigation software are mostly based on big data statistics. For safety reminders reported based on big data statistics, the data collected for remote roads is relatively small, and it is impossible to effectively predict, and it is not real-time. When facing emergencies, the driver needs to judge based on the actual road conditions at the time or his own driving experience. Therefore, the current safe driving reminder method for vehicle driving is still not intelligent and real-time enough. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a safe driving reminder method, device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a safe driving reminder method is provided for application in a mobile terminal, the method comprising:

[0006] Acquire an image outside the vehicle and identify a target object in the image; in response to identifying the target object and the target object being located in a preset area of ​​interest, monitor the moving trajectory of the target object and the moving trajectory of the vehicle; and provide safe driving reminders based on the moving trajectory of the target object and the moving trajectory of the vehicle.

[0007] In one implementation, monitoring the movement trajectory of the target object includes:

[0008] The multiple frames of images including the target object collected in real time are input into the optical flow detection model, and the movement direction and movement speed of the target object in the multiple frames of images are identified based on the optical flow detection model to determine the movement trajectory of the target object.

[0009] In one embodiment, monitoring the movement trajectory of the target object further includes:

[0010] The optical flow detection model extracts optical flow features of the target object in the multiple frames of images based on a pre-trained deep convolutional neural network, and performs residual processing on the optical flow features to identify the movement direction and speed of the target object in the multiple frames of images; or the optical flow detection model identifies the movement direction and speed of the target object in the multiple frames of images based on the constant brightness, temporal continuity and spatial consistency of the optical flow features of the target object.

[0011] In one implementation, monitoring the movement trajectory of the vehicle includes:

[0012] Based on data collected by an acceleration sensor in the terminal, a moving trajectory of the terminal is determined; and the moving trajectory of the terminal is determined as the moving trajectory of the vehicle.

[0013] In one implementation, the step of providing a safe driving reminder based on the moving trajectory of the target object and the moving trajectory of the vehicle includes:

[0014] If, based on the moving trajectory of the target object and the moving trajectory of the vehicle, it is determined that the target object is moving toward the vehicle, and no evasive movement of the vehicle in the opposite direction relative to the target object is detected, a safe driving reminder is issued.

[0015] In one embodiment, identifying a target object in the image includes:

[0016] The image is input into a target object detection network model; and a target object is identified in the image based on the target object detection network model. In one embodiment, before acquiring an image outside the vehicle and identifying the target object in the image, the method further includes:

[0017] Determine that conditions for enabling a safe driving reminder function are met; the conditions include at least one of the following: the terminal is located inside a vehicle; the terminal enables a navigation function.

[0018] According to a second aspect of an embodiment of the present disclosure, a safe driving reminder device is provided, the device comprising:

[0019] An identification unit is used to acquire an image outside the vehicle and identify a target object in the image; a monitoring unit is used to monitor the moving trajectory of the target object and the moving trajectory of the vehicle in response to identifying the target object and the target object being located in a preset area of ​​interest; and a reminder unit is used to provide safe driving reminders based on the moving trajectory of the target object and the moving trajectory of the vehicle.

[0020] In one implementation, the monitoring unit monitors the moving trajectory of the target object in the following manner:

[0021] The multiple frames of images including the target object collected in real time are input into the optical flow detection model, and the movement direction and movement speed of the target object in the multiple frames of images are identified based on the optical flow detection model to determine the movement trajectory of the target object.

[0022] In one implementation, the monitoring unit monitors the moving trajectory of the target object in the following manner:

[0023] The optical flow detection model extracts optical flow features of the target object in the multiple frames of images based on a pre-trained deep convolutional neural network, and performs residual processing on the optical flow features to identify the movement direction and speed of the target object in the multiple frames of images; or the optical flow detection model identifies the movement direction and speed of the target object in the multiple frames of images based on the constant brightness, temporal continuity and spatial consistency of the optical flow features of the target object.

[0024] In one implementation, the monitoring unit monitors the movement trajectory of the vehicle in the following manner:

[0025] Based on data collected by an acceleration sensor in the terminal, a moving trajectory of the terminal is determined; and the moving trajectory of the terminal is determined as the moving trajectory of the vehicle.

[0026] In one implementation, the reminder unit performs a safe driving reminder based on the moving trajectory of the target object and the moving trajectory of the vehicle in the following manner:

[0027] If, based on the moving trajectory of the target object and the moving trajectory of the vehicle, it is determined that the target object is moving toward the vehicle, and no evasive movement of the vehicle in the opposite direction relative to the target object is detected, a safe driving reminder is issued.

[0028] In one implementation, the recognition unit recognizes the target object in the image in the following manner:

[0029] The image is input into a target object detection network model; and a target object is identified in the image based on the target object detection network model.

[0030] In one embodiment, before acquiring an image outside the vehicle and identifying a target object in the image, the reminder unit is further configured to:

[0031] Determine that conditions for enabling a safe driving reminder function are met; the conditions include at least one of the following: the terminal is located inside a vehicle; the terminal enables a navigation function.

[0032] According to a third aspect of an embodiment of the present disclosure, a safe driving reminder device is provided, comprising:

[0033] A processor; a memory for storing processor executable instructions; wherein the processor is configured to: execute the safe driving reminder method described in the first aspect or any one of the embodiments of the first aspect.

[0034] According to the fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the safe driving reminder method described in the first aspect or any one of the embodiments of the first aspect.

[0035] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: the present disclosure realizes intelligent vehicle-machine linkage, and obtains target objects outside the vehicle in real time through a terminal placed on the vehicle. When the target object is in a preset area of ​​interest, safe driving reminders are given by comparing the moving trajectory of the target object and the vehicle.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] Figure 1 The figure is a flow chart of a safe driving reminder method according to an exemplary embodiment.

[0039] Figure 2 The figure is a schematic diagram of a driver's area of ​​interest in a safe driving reminder method according to an exemplary embodiment.

[0040] Figure 3 The figure is a flow chart of a safe driving reminder method according to an exemplary embodiment.

[0041] Figure 4 The figure is a schematic diagram of a target detection network of a safe driving reminder method according to an exemplary embodiment.

[0042] Figure 5 The figure is a flow chart of a safe driving reminder method according to an exemplary embodiment.

[0043] Figure 6 The figure is a schematic diagram of target optical flow detection of a safe driving reminder method according to an exemplary embodiment.

[0044] Figure 7 The figure is a flow chart of a safe driving reminder method according to an exemplary embodiment.

[0045] Figure 8 The figure is a block diagram of a safe driving reminder device according to an exemplary embodiment.

[0046] Fig. 9 The figure is a block diagram of a safe driving reminder device according to an exemplary embodiment.

[0047] Fig.10 The figure is a block diagram of a safe driving reminder device according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0049] With the development of smart terminals and navigation software, people will use terminal software for navigation while driving. However, the auxiliary safety driving reminders of terminal navigation software are mostly based on big data statistics. Navigation software can only report safety reminders for vehicles that appear multiple times on a certain road based on historical data statistics. Safety reminder reports based on big data statistics collect less data for remote roads, cannot be effectively predicted, and are not real-time. When faced with emergencies, drivers can only make judgments based on the actual road conditions at the time or their own driving experience.

[0050] Based on this, an embodiment of the present disclosure provides a safe driving reminder method, which obtains the target object outside the vehicle in real time through a terminal placed on the vehicle. When the target object is in a preset area of ​​interest, a safe driving reminder is given by comparing the moving trajectory of the target object and the vehicle.

[0051] The present invention effectively reduces the occurrence of traffic accidents and assists safe driving. By using the low-power image sensor of the terminal to obtain a larger field of view, it can provide safety reminders when the driver is tired, distracted, or in the driver's blind spots and areas that are difficult to notice. It realizes intelligent vehicle-machine linkage, can sense without wired connection, and has a wide range of applications.

[0052] The embodiments of the present disclosure will illustrate the safe driving reminder method in conjunction with the accompanying drawings.

[0053] It should be noted that the safe driving reminder involved in the embodiments of the present disclosure may be a safe driving reminder during vehicle driving, or may be a safe driving reminder during vehicle stationary, which is not limited in the embodiments of the present disclosure.

[0054] Figure 1 is a flow chart of a safe driving reminder method according to an exemplary embodiment. Figure 1 As shown, the safe driving reminder method is applied to a mobile terminal and includes the following steps.

[0055] In step S11 , an image outside the vehicle is acquired, and a target object is identified in the image.

[0056] In the disclosed embodiment, the method is applied to a mobile terminal, such as a mobile phone, a tablet or other terminal. The terminal is equipped with an image sensor, such as a binocular camera or a monocular camera. The image sensor of the terminal can be in the front direction, that is, the sensor can capture images in the area of ​​interest in front of the vehicle, and the image in the front can include people or vehicles. It is understandable that the vehicle can also be set with other image sensors with other directions of collection.

[0057] In the disclosed embodiment, the target object may be a vehicle, a pedestrian, an obstacle or the like, and the terminal on the vehicle collects images outside the vehicle in real time and identifies the target object.

[0058] In step S12, in response to identifying a target object and the target object being located in a preset region of interest, a moving trajectory of the target object and a moving trajectory of the vehicle are monitored.

[0059] In the embodiment of the present disclosure, the region of interest is a predefined region. Usually, the preset region of interest is a blind spot of the driver's visual field or an area that is not easy to notice during the driving of the vehicle.

[0060] In the present disclosure, Figure 2 , the preset area of ​​interest is the box area, that is, the driver's blind spot or the area that is not easy to notice. For example, during the driving of the vehicle, the driver usually observes the road conditions on both sides of the vehicle through the exterior rearview mirror. However, during the driving of the vehicle, when the vehicle turns sharply, the outline of the exterior rearview mirror and the A-pillar outline of the vehicle block a lot of the driver's sight. The A-pillar outline is the pillar between the front windshield and the left front door. When turning left, it will block the driver's sight, resulting in a large visual blind spot for the driver, causing the driver to be unable to clearly observe the road conditions on both sides of the vehicle when turning in time, which poses a great safety hazard. Therefore, this area can be set as the area of ​​interest. In this area, the moving trajectory of target objects such as vehicles or people can be monitored through the optical flow detection model, and the moving trajectory of the vehicle can be monitored through the inertial measurement unit (IMU) of the terminal.

[0061] In step S13, a safe driving reminder is given based on the moving trajectory of the target object and the moving trajectory of the vehicle.

[0062] In the disclosed embodiment, when the target object moves toward the vehicle and the vehicle does not make evasive movement in the opposite direction, a safe driving reminder is issued.

[0063] In the disclosed embodiment, the terminal placed on the vehicle can monitor the target object in the area of ​​interest in real time, and by comparing the moving trajectory of the target object and the vehicle, when the target object moves toward the vehicle and the vehicle does not make evasive movement in the opposite direction, a safe driving reminder is provided.

[0064] The following is an explanation of the safe driving reminder method involved in the present disclosure.

[0065] First, the process of identifying a target object in an image captured by a terminal according to an embodiment of the present disclosure is described.

[0066] In one implementation of the disclosed embodiment, target objects are identified based on a target object detection network.

[0067] In the embodiment of the present disclosure, before acquiring an image outside the vehicle and identifying a target object in the image, it is determined that the conditions for enabling the safe driving reminder function are met. In the embodiment of the present disclosure, the terminal is located inside the vehicle, and the terminal enables the navigation function, for example, it can be detected that the terminal meets the corresponding conditions to determine that the current scene requires a safe driving reminder.

[0068] Exemplarily, the terminal may be placed on the dashboard, that is, a position between the front windshield and the steering wheel of the vehicle.

[0069] Exemplarily, the terminal may be placed on a bracket on a dashboard to facilitate the acquisition of images of the vehicle's external environment with a wide viewing angle.

[0070] Exemplarily, in the embodiments of the present disclosure, the terminal can detect that it is in motion, or the terminal can detect that the navigation software is running, or the terminal can detect that it is in driving mode, and determine that the vehicle is currently in a scenario where a safe driving reminder is required.

[0071] Figure 3 is a flowchart of a method for identifying a target object in an image according to an exemplary embodiment. Figure 3 As shown, the method for identifying a target object in an image includes the following steps.

[0072] In step S21, the image collected by the terminal is input into the target object detection network model.

[0073] In step S22, the target object is identified in the image based on the target object detection network model.

[0074] Among them, the target object detection network involved in the embodiment of the present disclosure can be a trained YOLO model, and the image to be detected can be recognized and processed in the following manner: each frame of the image to be detected is input into the trained neural network recognition model, and the neural network recognition model outputs the recognition result of each frame of the image to be detected accordingly, wherein the recognition result includes whether there is a target object in the image to be detected, and the position of the target object in the image to be detected.

[0075] Among them, the neural network recognition model can be pre-trained in the following manner: first, at least one training image in the training image set is input into the neural network recognition model, and the neural network recognition model outputs the recognition result of the training image, wherein the training image has a label, and the label includes whether there is a target object in the training image, and the position of the target object in the training image; next, according to the recognition result of the training image and the label of the training image, the network loss value is determined; finally, according to the network loss value, the network parameters of the neural network recognition model are adjusted until the neural network recognition model converges.

[0076] Taking the target object as a vehicle as an example, a large number (at least tens of thousands) of vehicle images of different models and different angles can be used to form an image training set.

[0077] For example, the neural network recognition model can adopt the YOLO model. Figure 4 In this company, the YOLO network structure is trained through self-collected data sets or public data sets. A frame of image obtained by the terminal device is input to the YOLO model, and the YOLO model divides the image into a 7*7 grid. Each cell predicts multiple bounding boxes and the confidence of the bounding box.

[0078] The so-called bounding box is a rectangular box generated for the target object after it is found in the image. It is used to define the position and size of the target object. The position and size of the bounding box can be represented by 4 values: (x, y, w, h), where (x, y) is the center coordinate of the bounding box, and w and h are the width and height of the bounding box. Although a cell has multiple bounding boxes, it can only identify one object, so the cell needs to predict the category of the object, but the bounding box does not.

[0079] The confidence of the bounding box is the probability that the target object exists. The confidence score is obtained by calculating a standard (Intersection over Union, IoU) of the accuracy of the corresponding object between the predicted box and the actual box. The higher the IoU, the higher the confidence score. A threshold is set. When the confidence score of the predicted box is greater than or equal to the threshold, the box is considered a valid box and the result is output; when the confidence score of the predicted box is less than the threshold, it is considered an invalid box and discarded.

[0080] The final output structure is a three-dimensional matrix of 7*7*20, where 20 represents the types of objects that the recognizer can recognize, and the output 7*7 represents the probability that the detected classification exists in the coordinate system that divides the input image. When the probability exceeds the set threshold, the classification result is output. In the disclosed embodiment, the recognizer can detect the presence of a vehicle and the location of the vehicle in real time.

[0081] In the embodiments of the present disclosure, the target object identified by the target detection network may be a predefined object, for example, a person, an object, etc. In one embodiment, the target object may be a vehicle, for example, a truck, a bus, a truck whose size is greater than a threshold, etc.

[0082] When the target object is a vehicle, the driving track of the vehicle serving as the target object is determined.

[0083] In the disclosed embodiments, the benefits of network model-based recognition are: (1) It has a self-learning function. For example, when realizing image recognition, it is only necessary to input many different image templates and the corresponding results to be recognized into the artificial neural network first, and the network will slowly learn to recognize similar images through the self-learning function. The self-learning function is particularly important for prediction. It is expected that artificial neural network computers in the future will provide economic forecasts, market forecasts, and benefit forecasts for mankind, and their application prospects are very promising. (2) It has an associative storage function. This association can be achieved using the feedback network of the artificial neural network. (3) It has the ability to find the optimal solution at high speed. Finding the optimal solution for a complex problem often requires a large amount of calculation. By using a feedback artificial neural network designed for a certain problem and giving full play to the high-speed computing power of the computer, it is possible to find the optimal solution quickly.

[0084] The following is an explanation of the implementation process of monitoring the moving trajectory of the target object in the embodiment of the present disclosure.

[0085] Figure 5 is a flow chart of a safe driving reminder method according to an exemplary embodiment. Figure 5 As shown, monitoring the moving trajectory of the target object includes:

[0086] In step S31, multiple frames of images including the target object collected in real time are input into the optical flow detection model.

[0087] In step S32, the moving direction and moving speed of the target object in the multiple frames of images are identified based on the optical flow detection model to determine the moving trajectory of the target object.

[0088] In the disclosed embodiment, illustratively, a plurality of frames of images captured of a vehicle are input into an optical flow detection model, and the optical flow detection model determines the moving trajectory of the vehicle by identifying the moving direction and moving speed of the plurality of frames of images of the vehicle.

[0089] In one embodiment, the optical flow detection model extracts optical flow features of the target object in multiple frames of images based on a pre-trained deep convolutional neural network, and performs residual processing on the optical flow features to identify the movement direction and speed of the target object in the multiple frames of images.

[0090] In the disclosed embodiment, the model Flownet2.0 can be used as an example, with two consecutive images as input and the optical flow value of each pixel as output. The learning is performed by calculating the distance between the actual value and the predicted value of the optical flow to obtain the moving trajectory of the vehicle.

[0091] In another embodiment, the optical flow detection model identifies the movement direction and movement speed of the target object in multiple frames of images based on the constant brightness, temporal continuity and spatial consistency of the optical flow features of the target object.

[0092] In the embodiments of the present disclosure, the Lucas-Kanade algorithm can be used, which has the following three characteristics: (1) Constant brightness: that is, the brightness of the same point will not change with the change of time. This is the assumption of the basic optical flow method (all optical flow method variants must meet it) and is used to obtain the basic equation of the optical flow method. For grayscale images (the same applies to color images), this means that the grayscale value of the pixel will not change with the tracking of the frame. (2) Temporal continuity (small movement): that is, the change of time will not cause a drastic change in position, so that the grayscale can be used to calculate the partial derivative of the position (in other words, in the case of small movement, we can use the grayscale change caused by the unit position change between the previous and next frames to approximate the partial derivative of the grayscale with respect to the position). This is also an indispensable assumption of the optical flow method. The movement of the camera on the image changes slowly with time. In fact, this means that the change of time will not cause a drastic change in the pixel position, so that the grayscale value of the pixel can calculate the corresponding partial derivative with respect to the position. (3) Spatial consistency: adjacent points on the same surface in the scene have similar movements, and the distance between their projections on the image plane is also relatively close. Neighboring points on a scene are also neighboring points when projected onto an image, and their speeds are consistent. This is a unique assumption of the Lucas-Kanade optical flow method, because the basic equation of the optical flow method has only one constraint, and requires two unknown variables for the speeds in the x and y directions. We assume that the feature point moves similarly in the neighborhood, and we can combine n equations to obtain the speeds in the x and y directions (n ​​is the total number of points in the neighborhood of the feature point, including the feature point). The first two of the three basic assumptions are basic assumptions of the optical flow method, and the third is unique to the LK algorithm.

[0093] In the disclosed embodiment, illustratively, the moving direction and moving speed of the vehicle are identified by the Lucas-Kanade algorithm, so as to calculate the moving trajectory of the vehicle, laying a foundation for monitoring the moving trajectory of the target object below.

[0094] Figure 6 is a flow chart of a safe driving reminder method according to an exemplary embodiment. Figure 6 As shown, the movement trajectory of the vehicle is monitored, including:

[0095] In step S41, the movement trajectory of the terminal is determined based on data collected by an acceleration sensor in the terminal.

[0096] In the embodiments of the present disclosure, illustratively, an IMU module on a mobile phone may be used to collect data, and a three-axis accelerometer may be used to determine the motion state of the mobile phone, thereby determining the movement trajectory of the mobile phone.

[0097] In step S42, the moving trajectory of the terminal is determined as the moving trajectory of the vehicle.

[0098] In the disclosed embodiment, illustratively, the moving trajectory of the mobile phone is determined as the moving trajectory of the vehicle.

[0099] In the disclosed embodiment, the IMU module is very convenient for collecting motion data. At the same time, the data from the IMU is continuous and real-time, and can provide real-time feedback, so as to better determine the moving trajectory of the terminal and determine the moving trajectory of the terminal as the moving trajectory of the vehicle.

[0100] In the disclosed embodiment, a safe driving reminder is provided based on the moving trajectory of the target object and the moving trajectory of the vehicle. For example, when the moving trajectory of other vehicles deviates toward the direction of the vehicle and the terminal on the vehicle does not detect that the vehicle has avoided in the opposite direction, a safe driving reminder is provided.

[0101] In an exemplary embodiment of the present disclosure, multiple frames of images of a vehicle are input into an optical flow detection model, and the optical flow detection model identifies the movement direction and movement speed of the vehicle in the multiple frames of images to determine the movement trajectory of the vehicle, so as to achieve the following: Figure 7 For example, vehicle a and side vehicle b are driving in different lanes, and the optical flow detection model of the terminal on vehicle a predicts and supervises the driving trajectory of side vehicle b as follows: Figure 7 , there is no bias towards the moving trajectory of vehicle a. The optical flow detection model of vehicle a terminal supervises and predicts the driving trajectory of vehicle c as follows Figure 7 , with a trajectory that moves toward a.

[0102] In the disclosed embodiment, when providing a safe driving reminder, the reminder may be provided in the form of sending a prompt message.

[0103] Among them, in the embodiment of the present disclosure, the safe driving reminder can be a voice prompt, or it can be other information with a prompting effect. For example, it can be a voice message such as "pay attention to avoid" or it can be a beeping sound for safety reminder.

[0104] The safe driving reminder method provided by the disclosed embodiment collects images of the driver's blind spots or areas that are difficult to notice in real time through a mobile phone placed on the vehicle, identifies other vehicles in the image, monitors and predicts the movement trajectories of other vehicles through a pre-trained YOLO model, and determines the movement of the vehicle through the IMU module of the mobile phone. When other vehicles move toward the vehicle, if the mobile phone does not detect that the vehicle has an evasive movement in the opposite direction to the other vehicle, a safe driving reminder will be issued.

[0105] Based on the same concept, the embodiment of the present disclosure also provides a safe driving reminder device.

[0106] It is understandable that in order to achieve the above functions, the safe driving reminder device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0107] Figure 8 is a block diagram of a safe driving reminder device according to an exemplary embodiment. Figure 8 The device 100 includes an identification unit 101, a monitoring unit 102 and a reminder unit 103.

[0108] The recognition unit 101 acquires an image outside the vehicle and recognizes a target object in the image; the monitoring unit 102 is used to monitor the moving trajectory of the target object and the moving trajectory of the vehicle in response to the recognition of the target object and the target object being located in a preset area of ​​interest; the reminder unit 103 is used to provide safe driving reminders based on the moving trajectory of the target object and the moving trajectory of the vehicle.

[0109] In the embodiment of the present disclosure, the monitoring unit 102 monitors the moving trajectory of the target object in the following manner: multiple frames of images including the target object collected in real time are input into an optical flow detection model, and the movement direction and movement speed of the target object in the multiple frames of images are identified based on the optical flow detection model to determine the moving trajectory of the target object.

[0110] In the embodiment of the present disclosure, the monitoring unit 102 monitors the moving trajectory of the target object in the following manner: the optical flow detection model extracts the optical flow features of the target object in the multi-frame images based on the pre-trained deep convolutional neural network, and performs residual processing on the optical flow features to identify the movement direction and speed of the target object in the multi-frame images; or the optical flow detection model identifies the movement direction and speed of the target object in the multi-frame images based on the constant brightness, temporal continuity and spatial consistency of the optical flow features of the target object.

[0111] In the embodiment of the present disclosure, the monitoring unit 102 monitors the moving trajectory of the vehicle in the following manner: determining the moving trajectory of the terminal based on data collected by an acceleration sensor in the terminal; and determining the moving trajectory of the terminal as the moving trajectory of the vehicle.

[0112] In the embodiment of the present disclosure, the reminder unit 103 provides a safe driving reminder based on the moving trajectory of the target object and the moving trajectory of the vehicle in the following manner: if, based on the moving trajectory of the target object and the moving trajectory of the vehicle, it is determined that the target object is moving toward the vehicle, and no avoidance movement of the vehicle in the opposite direction relative to the target object is detected, a safe driving reminder is issued.

[0113] In the embodiment of the present disclosure, the recognition unit 101 recognizes the target object in the image in the following manner: inputting the image into a target object detection network model; and recognizing the target object in the image based on the target object detection network model.

[0114] In the embodiment of the present disclosure, the reminder unit 103 adopts the following method before acquiring the image outside the vehicle and identifying the target object in the image: determining that the conditions for turning on the safe driving reminder function are met; the conditions include at least one of the following: the terminal is located inside the vehicle and the terminal turns on the navigation function.

[0115] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0116] Fig. 9 2 is a block diagram of a device 200 for safe driving reminder according to an exemplary embodiment. The device 200 may be provided as a terminal. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0117] Reference Fig. 9, the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0118] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0119] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0120] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0121] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0122] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0123] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0124] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0125] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0126] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0127] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0128] Fig.10 is a block diagram of a device 300 for safe driving reminder according to an exemplary embodiment. For example, the device 300 may be provided as a server. Fig.10 , the apparatus 300 includes a processing component 322, which further includes one or more processors, and a memory resource represented by a memory 332 for storing instructions, such as an application, that can be executed by the processing component 322. The application stored in the memory 332 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 322 is configured to execute the instructions to perform the above method.

[0129] The device 300 may also include a power supply component 326 configured to perform power management of the device 300, a wired or wireless network interface 350 configured to connect the device 300 to a network, and an input / output (I / O) interface 358. The device 300 may operate based on an operating system stored in the memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0130] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 332 including instructions, which can be executed by the processing component 322 of the device 300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0131] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0132] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0133] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0134] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0135] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0136] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A safe driving reminder method, characterized in that: The method is applied to a mobile terminal, and the method comprises: acquiring an image outside the vehicle and identifying a target object in the image; In response to identifying a target object, and the target object being located in a preset area of ​​interest, monitoring a moving trajectory of the target object and a moving trajectory of the vehicle; Based on the moving trajectory of the target object and the moving trajectory of the vehicle, a safe driving reminder is provided.

2. The safe driving reminder method according to claim 1, characterized in that: Monitoring the movement trajectory of the target object includes: The multiple frames of images including the target object collected in real time are input into the optical flow detection model, and the movement direction and movement speed of the target object in the multiple frames of images are identified based on the optical flow detection model to determine the movement trajectory of the target object.

3. The safe driving reminder method according to claim 2, characterized in that: The method further comprises: The optical flow detection model extracts optical flow features of the target object in the multiple frames of images based on a pre-trained deep convolutional neural network, and performs residual processing on the optical flow features to identify the movement direction and movement speed of the target object in the multiple frames of images; or The optical flow detection model identifies the moving direction and moving speed of the target object in the multiple frames of images based on the constant brightness, temporal continuity and spatial consistency of the optical flow features of the target object.

4. The method according to claim 1, characterized in that: The monitoring of the movement trajectory of the vehicle includes: Determining a moving trajectory of the terminal based on data collected by an acceleration sensor in the terminal; The moving trajectory of the terminal is determined as the moving trajectory of the vehicle.

5. The safe driving reminder method according to any one of claims 1 to 3, characterized in that: The providing a safe driving reminder based on the moving trajectory of the target object and the moving trajectory of the vehicle includes: If, based on the moving trajectory of the target object and the moving trajectory of the vehicle, it is determined that the target object is moving toward the vehicle, and no evasive movement of the vehicle in the opposite direction relative to the target object is detected, a safe driving reminder is issued.

6. The safe driving reminder method according to claim 1, characterized in that: The identifying a target object in the image comprises: The image is input into a target object detection network model, and a target object is identified in the image based on the target object detection network model.

7. The safe driving reminder method according to claim 1, characterized in that: Before acquiring an image outside the vehicle and identifying the target object in the image, the method further includes: Determine whether the conditions for enabling the safe driving reminder function are met; The conditions include at least one of the following: The terminal is located inside the vehicle; The terminal starts the navigation function.

8. A safe driving reminder device, characterized in that: include: a recognition unit, configured to acquire an image outside the vehicle and recognize a target object in the image; A monitoring unit, configured to monitor a moving trajectory of the target object and a moving trajectory of the vehicle in response to identifying the target object and the target object being located in a preset area of ​​interest; The reminder unit is used to provide a safe driving reminder based on the moving trajectory of the target object and the moving trajectory of the vehicle.

9. A safe driving reminder device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the safe driving reminder method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the safe driving reminder method described in any one of claims 1 to 7.

Images