Rearview mirror based on streaming media and driving assistance method thereof

Abstract

According to the rearview mirror based on the streaming media and the driving assistance method thereof, the vehicle image is obtained after the environment information behind the vehicle is obtained through the camera, and the vehicle image is clearly presented through the streaming media display screen, so that a driver can obtain the clear and effective environment information behind the vehicle without turning the head or depending on the limited view of a traditional rearview mirror; and auxiliary driving is facilitated. Moreover, the distance between objects behind the vehicle is accurately recognized through a distance measurement algorithm and displayed on a streaming media rearview mirror display screen or output through an output device, so that a driver can conveniently obtain the rear distance information without turning the head or depending on the limited view of a traditional rearview mirror, the vehicle distance can be more accurately judged, assistance in the aspect of driving safety is provided, and the driving safety is improved. Traffic accidents caused by view blind areas or distance misjudgment are effectively reduced, and driving safety and convenience are remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of streaming media rearview mirrors, and in particular to a streaming media-based rearview mirror and a driving assistance method thereof. Background Art

[0002] With the improvement of vehicle sensor technology and video transmission technology, streaming media rearview mirrors are being used in more and more vehicles. The so-called streaming media rearview mirror uses a rear camera to shoot the rear view of the vehicle in real time, and transmits the video through a coaxial cable to present it losslessly and without delay on the central rearview mirror display screen, providing a wider, clearer and adjustable rear view, so that the driver can accurately understand the situation behind the vehicle.

[0003] The current application scope of streaming media rearview mirrors is relatively limited. Its main function is to simply transmit and display the real-time images captured by the camera on the display screen of the central rearview mirror. In the key aspects of vehicle assisted driving, it has many shortcomings. For example, it is difficult to accurately judge the distance between the vehicle in the image and the current vehicle through the screen or image displayed on the display screen of the streaming media rearview mirror. At present, it can only be estimated based on experience. Once the judgment is wrong, it is easy to cause a safety accident.

[0004] Therefore, the current streaming media rearview mirror cannot substantially help the driver to control the vehicle more safely and efficiently, which greatly limits its value in improving driving experience and ensuring driving safety. Summary of the invention

[0005] In order to overcome the problems existing in the related art, this specification provides a streaming media-based rearview mirror and a driving assistance method thereof.

[0006] According to a first aspect of an embodiment of this specification, there is provided an assisted driving method based on a streaming media rearview mirror, wherein the streaming media rearview mirror comprises a camera and an interior rearview mirror comprising a display screen, and the method comprises:

[0007] Acquire a vehicle image obtained by collecting environmental information behind the vehicle through the camera; the environmental information includes the photographed object;

[0008] Determine the distance information of the object relative to the vehicle based on a preset distance measurement algorithm;

[0009] The vehicle image is displayed on the display screen and the distance information is outputted through an output device for viewing by a driver.

[0010] According to an assisted driving method based on a streaming media rearview mirror provided by the present application, after determining the distance information of the object relative to the vehicle based on a preset distance measurement algorithm, the method further includes:

[0011] Predicting a first motion trajectory of the vehicle according to first speed information of the vehicle and vehicle driving data;

[0012] Determining second speed information and a second motion trajectory of the object based on the distance information and the interpolation algorithm;

[0013] analyzing a collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory;

[0014] The vehicle image is displayed on the display screen and the collision risk is outputted through an output device for viewing by a driver.

[0015] According to an assisted driving method based on a streaming media rearview mirror provided by the present application, after analyzing the collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory, the method further includes:

[0016] When the vehicle is in a state of turning or changing lanes, generating a torque request according to current vehicle driving data of the vehicle and the collision risk;

[0017] A torque request is sent to a controller of the electric power steering system of the vehicle through a CAN interface, so that the electric power steering system responds to the torque request and warns the driver.

[0018] According to an assisted driving method based on a streaming media rearview mirror provided by the present application, the electric power steering system includes a steering wheel, the torque request includes a vibration frequency and an amplitude,

[0019] The method further comprises:

[0020] A torque request is sent to a controller of the electric power steering system of the vehicle through a CAN interface, so that the electric power steering system responds to the torque request and controls the steering wheel to vibrate and warn with the received vibration frequency and amplitude.

[0021] According to an assisted driving method based on a streaming media rearview mirror provided by the present application, after analyzing the collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory, the method further includes:

[0022] When the vehicle is in a reversing state, generating a braking request according to current vehicle driving data of the vehicle and the collision risk;

[0023] A torque request is sent to the braking system of the vehicle through the CAN interface, so that the braking system responds to the braking request and performs auxiliary braking on the vehicle.

[0024] According to an assisted driving method based on a streaming media rearview mirror provided in the present application, the interior rearview mirror includes a lighting control circuit,

[0025] The method further comprises:

[0026] When the vehicle is in a reversing state, a control instruction is sent to a light control circuit of the interior rearview mirror to control at least one light area of ​​the interior rearview mirror to light up.

[0027] According to an assisted driving method based on a streaming media rearview mirror provided in the present application, the output device includes at least one of a display screen and a voice device.

[0028] According to an assisted driving method based on a streaming media rearview mirror provided by the present application, the display screen displays the vehicle image and the output device outputs the distance information, including:

[0029] Based on the display mode corresponding to the distance information, the vehicle image and the distance information are displayed on the display screen, and the display mode is switched according to the degree of change of the distance information.

[0030] According to an assisted driving method based on a streaming media rearview mirror provided by the present application, the display mode includes a layered display mode.

[0031] The displaying the vehicle image and the distance information through the display screen includes:

[0032] Acquire the vehicle image as a base layer;

[0033] The distance information is superimposed on the corresponding position of the vehicle in the basic layer in the form of annotation, and the transparency of the annotated distance information decreases as the distance increases.

[0034] According to an assisted driving method based on a streaming media rearview mirror provided by the present application, the distance information of the object relative to the vehicle is determined based on a preset distance measurement algorithm, including:

[0035] Detecting the vehicle image, and determining the road plane coordinates of the object in the coordinate system of the vehicle based on the internal and external parameters of the camera;

[0036] The distance information of the object relative to the vehicle is determined by a preset distance measurement algorithm and the road plane coordinates.

[0037] According to an assisted driving method based on a streaming media rearview mirror provided in the present application, the method further includes:

[0038] Obtaining current posture information of the camera;

[0039] The internal and external parameters of the camera are updated according to the posture information.

[0040] The present application also provides a streaming media rearview mirror, which includes a camera and an interior rearview mirror including a display screen; the camera includes a memory, a processor, and an assisted driving program based on a streaming media rearview mirror stored on the memory and running on the processor, and when the processor executes the assisted driving program based on the streaming media rearview mirror, the steps of the assisted driving method based on the streaming media rearview mirror as described in any one of the above are implemented.

[0041] The present application also provides a streaming media rearview mirror, which includes a camera and an interior rearview mirror including a display screen; the interior rearview mirror includes a memory, a processor, and an assisted driving program based on a streaming media rearview mirror stored on the memory and running on the processor, and when the processor executes the assisted driving program based on the streaming media rearview mirror, the steps of the assisted driving method based on the streaming media rearview mirror as described in any one of the above are implemented.

[0042] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the assisted driving method based on a streaming media rearview mirror as described in any one of the above is implemented.

[0043] In the embodiments of this specification, based on the streaming media rearview mirror and its driving assistance method, the vehicle image obtained after the environmental information behind the vehicle is obtained through the camera is clearly presented through the streaming media display screen, so that the driver does not need to turn his head or rely on the limited field of view of the traditional rearview mirror to obtain clear and effective environmental information behind the vehicle, which is helpful for assisted driving. In addition, the distance between objects behind the vehicle is accurately identified through the distance measurement algorithm and presented on the streaming media rearview mirror display screen or output through the output device, so that the driver can easily obtain the rear distance information without turning his head or relying on the limited field of view of the traditional rearview mirror, so that the distance between vehicles can be judged more accurately, providing assistance in driving safety, effectively reducing traffic accidents caused by blind spots in vision or misjudgment of distance, and significantly improving the safety and convenience of driving.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0046] Figure 1 is a structural diagram of a streaming media rearview mirror according to an exemplary embodiment of the present specification;

[0047] Figure 2 is a flowchart of an assisted driving method based on a streaming media rearview mirror according to an exemplary embodiment of this specification;

[0048] Figure 3 is a schematic diagram of a coordinate system according to an exemplary embodiment of this specification;

[0049] Figure 4 is a schematic diagram of a lens imaging according to an exemplary embodiment of this specification;

[0050] Figure 5 is a coordinate conversion schematic diagram shown in this specification according to an exemplary embodiment;

[0051] Figure 6 is another lens imaging schematic diagram shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0052] Here, the technical solutions in the embodiments (or "implementations") of the present application will be described clearly and completely in conjunction with the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0053] If there are terms involving directional indications or positional relationships in the embodiments of the present application (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings); if the specific posture changes, the directional indication or positional relationship will also change accordingly. In addition, the terms "first", "second", etc. involved in the embodiments of the present application are only used for the purpose of convenience of description and cannot be understood as indicating or implying relative importance.

[0054] The present application provides a streaming media-based rearview mirror and a driving assistance method thereof. The present application is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0055] The current application scope of streaming media rearview mirrors is relatively limited. Its main function is to simply transmit and display the real-time images captured by the camera on the display screen of the central rearview mirror. In the key aspects of vehicle assisted driving, it has many shortcomings. For example, it is difficult to accurately judge the distance between the vehicle in the image and the current vehicle through the screen or image displayed on the display screen of the streaming media rearview mirror. At present, it can only be estimated based on experience. Once the judgment is wrong, it is easy to cause a safety accident.

[0056] Therefore, the current streaming media rearview mirror cannot substantially help the driver to control the vehicle more safely and efficiently, which greatly limits its value in improving driving experience and ensuring driving safety.

[0057] In order to solve the above technical problems, this specification provides an assisted driving method based on a streaming media rearview mirror.

[0058] The distance between objects behind the vehicle is accurately identified through a distance-measuring algorithm and displayed on the streaming media rearview mirror display screen, providing assistance in driving safety. The driver can easily obtain rear distance information without turning his head or relying on the limited field of view of traditional rearview mirrors, thereby being able to judge the distance between vehicles more accurately, effectively reducing traffic accidents caused by blind spots or misjudgment of distance, and significantly improving driving safety and convenience.

[0059] It should be noted that the streaming media rearview mirror described in this article includes a camera and an interior rearview mirror including a display screen, and the assisted driving method based on the streaming media rearview mirror can be applied to the camera or the interior rearview mirror.

[0060] As an example, Figure 1 1 is a structural diagram of a streaming media rearview mirror provided in an embodiment of the present specification. Taking the application of the assisted driving method based on the streaming media rearview mirror to the interior rearview mirror as an example, the streaming media rearview mirror includes a camera and an interior rearview mirror including a display screen.

[0061] The camera is installed at the rear of the vehicle and is used for image input and processing, that is, collecting environmental information behind the vehicle in real time to obtain a vehicle image.

[0062] The camera includes a lens group, an image sensor, and a digital signal processing chip. The lens group captures the optical image of the road scene and projects it onto the image sensor. The image sensor converts the optical signal into an electrical signal, which is then converted into a digital image signal after A / D analog-to-digital conversion. The DSP pre-processes the digital image signal, including framing, color correction, white balance adjustment, etc.

[0063] The interior rearview mirror is connected to the camera via a CAN line and is used to receive the vehicle image obtained after the camera collects the environmental information behind the vehicle. The interior rearview mirror includes a display screen and a main control chip. The main control chip is used for feature extraction and target recognition.

[0064] Specifically, based on the preprocessing of the image, the system extracts feature points in the image, such as edges, textures, etc. Machine learning or neural network algorithms are used to classify and identify feature data to distinguish different objects (such as vehicles, pedestrians, traffic signs, etc.). Furthermore, the main control chip uses distance measurement and speed measurement algorithms to obtain the distance between the object and the vehicle and the predicted motion trajectory of other vehicles for subsequent risk reminders. The information calculated by the main control chip is displayed on the display screen for assisted driving.

[0065] The above-mentioned streaming media rearview mirror also includes a CAN interface, which is used to communicate with the controllers of various systems / modules of the vehicle, so as to realize the interactive transmission of various vehicle signals and information of the rearview mirror inside the vehicle, and clearly display them through the display screen, such as visual prompts, auditory prompts, etc. These functions provide the driver with visual and auditory prompts, assisted braking, assisted lane keeping, etc., to improve driving safety.

[0066] Figure 2 1 is a flowchart of an assisted driving method based on a streaming media rearview mirror provided in an embodiment of this specification, including the following steps 100 to 300:

[0067] Step 100, obtaining a vehicle image obtained by collecting environmental information behind the vehicle through the camera; the environmental information includes the photographed object.

[0068] When the vehicle is started or the streaming media rearview mirror system is activated, the camera installed at a specific location on the rear of the vehicle (such as the inside of the rear windshield, near the rear license plate, etc.) starts working. The camera lens focuses on the area behind the vehicle, captures light information according to its set shooting parameters (such as focal length, aperture, frame rate, etc.), and converts it into electrical signals through the image sensor. For example, the camera's image sensor (such as a CMOS or CCD sensor) receives light and converts the light signal into a digital signal. These digital signals represent the brightness, color and other information of each pixel in the scene behind the vehicle, thereby initially forming the original image data behind the vehicle.

[0069] The collected raw image data is pre-processed, including but not limited to framing, color correction, white balance adjustment, etc., to obtain the vehicle image sent to the rearview mirror inside the vehicle. The vehicle image includes the photographed object, which refers to the information such as vehicles, pedestrians, traffic signs, etc. that are identified after analyzing the preliminarily processed image using the image processing algorithm. This information may affect the driving, reversing, turning, changing lanes, etc. of the vehicle.

[0070] Step 200: Determine the distance information of the object relative to the vehicle based on a preset distance measurement algorithm.

[0071] Based on preprocessing, the main control chip extracts the feature points of the vehicle image, and uses machine learning or neural network algorithms to classify and identify the feature data to distinguish different objects (such as vehicles, pedestrians, traffic signs, etc.).

[0072] As an example, edge detection, contour recognition and other operations are performed to distinguish different objects and backgrounds in the image. For example, by identifying the outline of the vehicle, the shape characteristics of pedestrians, the specific pattern of traffic signs, etc., the existence and position of the photographed object can be determined. The information of these identified vehicles, pedestrians, traffic signs, etc., as well as their position, size, motion state and other data in the image together constitute the environmental information containing the photographed object, providing basic data support for subsequent driving assistance functions (such as early warning, motion trajectory prediction, etc.).

[0073] At present, it is difficult to accurately judge the distance between the vehicle in the image and the current vehicle through the screen or image presented on the display screen of the streaming media rearview mirror. At present, it can only be estimated based on experience. Once the judgment is wrong, it is easy to cause a safety accident. The present invention uses a distance measurement algorithm to accurately identify the distance between the object behind the vehicle and the vehicle, and presents it on the display screen of the streaming media rearview mirror to solve the current problem.

[0074] In some embodiments, step 200 may specifically include the following steps 210 to 211:

[0075] Step 210 , detecting the vehicle image, and determining the road plane coordinates of the object in the coordinate system of the vehicle based on the internal and external parameters of the camera.

[0076] The vehicle image is detected to identify objects behind the vehicle that may affect the vehicle's driving (including but not limited to reversing, changing lanes, turning, etc.). For example, advanced computer vision algorithms are used for object detection, such as deep learning-based target detection algorithms, to identify vehicles, pedestrians, traffic signs, and other objects.

[0077] When an object in a vehicle image is detected, the location and range of the object are determined and marked with a polygonal box. The polygonal box is usually determined based on the boundary information of the object and is implemented by calculating the minimum circumscribed rectangle of the object. The polygonal box can be, but is not limited to, a rectangular box or a polygonal box that is consistent with the outer contour of the object.

[0078] In the two-dimensional coordinate system of the vehicle image, each pixel has its corresponding coordinate position. By extracting the boundary information of the polygonal box, the pixel coordinates of the boundary points of the polygonal box are obtained. These coordinates are used as the basic data for the subsequent geometric relationship derivation and are used to calculate the position information of the object on the actual road plane. It should be noted that the number of pixel coordinates of the boundary points of the polygonal box to be extracted can be determined according to actual needs, and can be determined based on but not limited to different needs such as recognition accuracy, computing efficiency, computing power consumption, positional relationship accuracy, and content richness.

[0079] Preferably, the two pixel coordinates of the bottom edge of the rectangular frame are extracted. This is because in many cases, especially for objects such as vehicles and pedestrians that have a certain height on the road plane and whose bottoms are in contact with the ground, the bottom edge coordinates can intuitively reflect their projection position on the road plane. Through these two endpoint coordinates combined with the corresponding geometric relationship derivation method, the actual position coordinates of the object on the road plane can be quickly calculated. In the subsequent calculation of the distance between objects, judgment of whether objects are in the same lane or whether there is a risk of collision, and other analyses involving the relative position relationship of the road plane, the information contained in the two endpoint coordinates of the bottom edge is often sufficient. At the same time, compared with extracting more boundary point coordinates, it can provide key data for subsequent processing procedures more concisely and efficiently.

[0080] Specifically, the pixel coordinates of the points in the rectangular frame representing the contour of the object are determined by combining the information of the rectangular frame, which are recorded as (m1, n1) and (m2, n2). Using the geometric relationship derivation method, the road plane coordinates (x1, y1) and (x2, y2) in the coordinate system where the vehicle is located are derived from the pixel plane coordinate points (m1, n1) and (m2, n2) of the image. Among them, the coordinate points are projected onto the ground, and the z axis is 0.

[0081] As an example, the process of using the geometric relationship derivation method to derive the road plane coordinates of the vehicle's coordinate system from the pixel plane coordinate points (m1, n1) and (m2, n2) of the image involves the conversion of the coordinate system. That is, the image processing process involves the calibration of internal and external parameters, that is, converting the image information captured by the camera into the road plane coordinate system we are familiar with.

[0082] Specifically, refer to Figure 3 , Figure 4 As shown, assuming that a point P in the world coordinate system v(X v ,Y v ,Z v ), the corresponding camera coordinate system coordinates are set to: P c (X c ,Y c ,Z c ), after the lens imaging, the corresponding point is Pi(Xi,Yi). The camera external parameter matrix is ​​the affine transformation matrix that transforms the world coordinates into the camera coordinates. The internal matrix is ​​the rotation matrix R and the translation matrix T. The world coordinate origin is coincident with the camera coordinate origin through rotation and translation. As shown in the following expression (1):

[0083]

[0084] Reference Figure 5 , the internal parameter matrix is ​​the affine transformation matrix that transforms the camera coordinates into the image coordinates. Let the focal length of the lens be f, and fx and fy in the x and y directions respectively. Finally, the image displayed on the screen uses the pixel coordinate system, with its origin in the upper left corner, the original imaging optical center at (m0, n0), the x-axis direction is specified as rightward, and the y-axis direction is specified as downward. Directly performing a direct transformation from world coordinates to pixel coordinates, the following expression (2) is obtained:

[0085]

[0086] Let K1 represent the intrinsic parameter and K2 represent the extrinsic parameter, then the coordinate mapping from a point in the world coordinate system to the pixel coordinate system is as follows:

[0087]

[0088] Conversely, for a pixel on the image finally obtained by the visual sensor, its pixel coordinates are converted into world coordinates, which can be solved by the following expression:

[0089]

[0090] Expand the above formula, and we get:

[0091]

[0092] Where R, K, and T are all known values. Rewrite the variables in the above equation as follows:

[0093]

[0094] Mat2=R -1 T (7)

[0095] Then we have:

[0096] Z v =Zc ×Mat1(2,0)-Mat2(2,0) (8)

[0097]

[0098] Since the target object is on the ground, Z v =0, we can get Z c , which can be substituted into expression (4) to solve.

[0099] Through the above process, the road plane coordinates of the object in the vehicle coordinate system in the world coordinate system are determined.

[0100] In some other embodiments, the method further comprises:

[0101] Step 220, obtaining the current posture information of the camera;

[0102] Step 221: Update the internal and external parameters of the camera according to the posture information.

[0103] There are bound to be bumps during driving, which causes the angle of the visual sensor relative to the road to change dynamically, causing the parameters of the camera geometry model mentioned above to change. If the ranging algorithm does not update this parameter change in time, an incorrect distance estimate will occur. The road is not always flat, it can tilt, rise or fall, so the direction of the car and the camera is constantly changing relative to the flat road image sequence. Correct external parameter calibration is essential for good vehicle ranging.

[0104] Therefore, when performing visual ranging, it is necessary to consider the pitch angle of the camera and promptly correct the parameter matrix containing this angle change to ensure the accuracy of the measurement.

[0105] Specifically, refer to Figure 6 The camera external parameters include a rotation matrix R, which describes how the camera rotates. When the camera pitch angle changes due to vehicle bumps, a new rotation matrix R needs to be found. cr To update the camera's external parameters, the movement of the vanishing point can reflect the change of the rotation matrix. In the three-dimensional world, lane lines are mostly parallel. However, from the images of tracks or road lane lines taken by the camera, it can be observed that the track lines or lane lines in the image are not parallel. These lines will intersect in the image. The intersection point is called the vanishing point.

[0106] Assuming that the car's forward direction is aligned with the lane line, the lane line intersects at Z = ∞, and (X, Y, Z) is the coordinate of a point in the world coordinate system. Combined with the camera geometry model mentioned above, the coordinate system has the following transformation:

[0107]

[0108] Since this is a homogeneous coordinate equation, we can multiply both sides by And incorporate this number into the lambda coefficient on the left:

[0109]

[0110] Since the Z of the vanishing point can be infinite, the pixel coordinate expression corresponding to the vanishing point is:

[0111]

[0112] Expanding the external parameter matrix, the above formula can be rewritten as:

[0113]

[0114] After simplification, we have:

[0115] λp ∞ =Kγ3 (14)

[0116] Note that each column of the rotation matrix is ​​a unit vector with a modulus of 1, so ||r3||=1, which means that λ is equal to

[0117]

[0118] Then we can deduce the value of γ3:

[0119]

[0120] Since the camera rotation matrix is ​​represented by roll, pitch, and yaw matrices, we have

[0121]

[0122] So we get

[0123] α=sin -1 R yz (18)

[0124]

[0125] After obtaining the new pitch angle, the rotation matrix R can be updated accordingly. cr As long as the vanishing point of each image frame can be determined, the camera external parameters can be corrected and the updated rotation matrix R cr It can be used to measure the distance of other targets in the image frame.

[0126] Through this embodiment, the internal and external parameters of the camera can be dynamically updated according to the real-time posture information of the camera on the vehicle, and the key factor of the real-time posture change of the camera can be fully considered in the actual calculation process, and the imaging deviation caused by camera shaking, steering, etc. can be effectively eliminated. The accuracy and reliability of the camera data can be significantly improved, and the accuracy of the distance information calculated subsequently can be improved, thereby providing more accurate and real-time image information for the vehicle's intelligent driving assistance system, effectively ensuring driving safety and the effective implementation of various intelligent driving functions.

[0127] Step 211, determining the distance information of the object relative to the vehicle through a preset distance measurement algorithm and the road plane coordinates.

[0128] The second coordinate of the vehicle specifies the position of the vehicle in the world coordinate system, which is equivalent to a reference point and provides a reference for measuring the distance of the object relative to the vehicle. By coordinating with the road plane coordinates of the object in the coordinate system and the preset distance measurement algorithm, the position of the object can be converted to the same coordinate system as the vehicle for distance calculation.

[0129] As an example, the distance measurement algorithm includes a difference frame algorithm and a Euclidean distance.

[0130] The frame difference algorithm described in this article is a method for calculating the speed of an object based on the difference between frames in a video sequence. It is based on a simple assumption: the greater the change in the position of an object between adjacent frames, the faster the object's speed. The principle is to calculate the position difference of an object between two adjacent frames, and then calculate the speed of the object through the time interval.

[0131] Assume that the road plane coordinates of the object in the vehicle image in the tth frame and the (t-1th frame are P t and P t-1 The plane coordinates of the road are P t and P t-1 It is obtained by the above geometric derivation method. Then, the Euclidean distance or other similarity measurement methods are used to calculate the distance d between the object and the vehicle in the vehicle image.

[0132] Specifically, calculate P t and P t-1 Distance between:

[0133] d=||P t -P t-1 ||

[0134] The distance information d of the object relative to the vehicle is determined through the above embodiment. The distance information effectively reduces traffic accidents caused by blind spots or distance misjudgment, and significantly improves driving safety and convenience.

[0135] Step 300: display the vehicle image via the display screen and output the distance information via an output device for the driver to view.

[0136] The vehicle image is sent to the display screen of the interior rearview mirror, where it is displayed for the driver's reference. At the same time, the distance information is first transmitted to the corresponding output control unit.

[0137] As an example, the output device includes at least one of a display screen and a voice device.

[0138] This means that if the output device is a voice device mounted on the vehicle or in the rearview mirror, the distance information is played through the voice device. The vehicle image and distance information are presented to the driver through the display screen and the output device respectively, so that the driver can intuitively obtain the visual image of the vehicle appearance and surrounding conditions, and accurately know the distance between the vehicle and surrounding objects through clear and unambiguous voice or other forms of distance information output, avoiding misjudgment of distance caused by visual information overload, and effectively improving the comprehensiveness and accuracy of the driver's perception of the vehicle's environment, thereby enhancing driving safety and control accuracy.

[0139] If the output device is the dashboard display or a dedicated head-up display (HUD) system in the car, the output control unit will encode the distance information in a specific format (such as a digital display format or a graphical display format). For the digital display format, the distance value is directly displayed in a specified area in digital form. If the output device is the display screen of the interior rearview mirror, the distance information and the vehicle image are displayed together on the display screen of the interior rearview mirror, so that the driver can view the environment behind the vehicle through the vehicle image and obtain the distance between the object in the rear environment and the vehicle, so as to assist the driver in making driving decisions, such as judging whether it is safe to change lanes and maintain an appropriate following distance from the vehicle behind.

[0140] In some embodiments, displaying the vehicle image through the display screen and outputting the distance information through an output device include:

[0141] Based on the display mode corresponding to the distance information, the vehicle image and the distance information are displayed on the display screen, and the display mode is switched according to the degree of change of the distance information.

[0142] When the output device is a display screen, the vehicle image and distance information are displayed simultaneously based on a preset display mode. In this embodiment, the vehicle image and distance information are displayed on the display screen based on the display mode corresponding to the distance information, and the display mode can be automatically switched as the distance information changes, so that the driver can perceive the distance between the vehicle and the target object more intuitively, clearly and dynamically, greatly enhancing the effectiveness and timeliness of information transmission, effectively assisting the driver to make quick and accurate driving decisions, and significantly improving driving safety and convenience.

[0143] In some other embodiments, the display mode includes a layered display mode, and the displaying of the vehicle image and the distance information through the display screen includes:

[0144] Acquire the vehicle image as a base layer;

[0145] The distance information is superimposed on the corresponding position of the vehicle in the basic layer in the form of annotation, and the transparency of the annotated distance information decreases as the distance information increases.

[0146] The vehicle image is used as the base layer to fully present the vehicle's appearance, detailed texture and other visual information, so that it occupies the main display area of ​​the display screen. For the depth information layer, it is constructed using translucent dynamic lines or geometric elements with specific colors. When displaying distance information, the distance information is superimposed on the corresponding position of the vehicle in the base layer in the form of annotations. For example, the elements representing the depth information are laid out in a manner that surrounds the periphery of the vehicle image or is distributed along the contours of key parts of the vehicle. As an example, the edge of the vehicle is displayed in the form of a contour frame connected by equally spaced dotted lines. The thickness, brightness or flashing frequency of the frame line changes dynamically according to the depth value. The greater the depth, the thicker and brighter the frame line or the higher the flashing frequency.

[0147] When the depth information element overlaps with the vehicle image, by adjusting its transparency, the vehicle image can still be clearly visible, and the depth information can also be intuitively perceived by the user without causing substantial occlusion to the vehicle image. At the same time, the transparency is adjusted to decrease as the distance increases. It can be understood that the smaller the distance, the greater the risk of collision between the object and the vehicle, and the driver needs to be reminded to focus on avoiding objects with higher risks. In this way, the vehicle image and depth information can be displayed clearly and without occlusion at the same time.

[0148] Through this embodiment, a layered display mode is adopted, which allows the driver to view the overall appearance of the vehicle while clearly and intuitively knowing the distance between each part and surrounding objects based on the changes in the transparency of the markings, thereby avoiding information clutter and interference and improving driving safety.

[0149] In other embodiments, the display mode also includes an icon display mode.

[0150] Add icons representing depth information at key locations in the vehicle image (such as the four corners of the vehicle, front and rear bumpers, etc.). These icons can be simple numbers showing the specific distance to objects around the vehicle, or arrows, where the length or thickness of the arrows indicates the distance, and the direction indicates the position of the object relative to the vehicle. This method can provide depth information concisely and clearly without interfering too much with the display of the vehicle image.

[0151] Based on the above first embodiment, a second embodiment of an assisted driving method based on a streaming media rearview mirror is proposed.

[0152] After determining the distance information of the object relative to the vehicle based on the preset distance measurement algorithm, the method further includes:

[0153] Step a1, predicting a first motion trajectory of the vehicle according to first speed information of the vehicle and vehicle driving data;

[0154] Step a2, determining the second speed information and the second motion trajectory of the object based on the distance information and the interpolation algorithm;

[0155] Step a3, analyzing the collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory;

[0156] Step a4, displaying the vehicle image through the display screen and outputting the collision risk through an output device for the driver to view.

[0157] The current first speed information and vehicle driving data of the vehicle are obtained. The vehicle driving data includes but is not limited to data of the electric power steering system, data of the power system, historical driving paths of the vehicle, etc. These data are integrated and a kinematic model and trajectory prediction algorithm are used to determine the possible driving path of the vehicle in the future to obtain a first motion trajectory.

[0158] Based on the aforementioned distance information and interpolation algorithm, after determining the distance information, the average speed v of the object is calculated by the time interval △t:

[0159] v=d / △t

[0160] Wherein, △t represents the time interval between the tth frame and the (t-1)th frame. In summary, the magnitude and direction of the second velocity information v of the object relative to the vehicle are obtained, and the second motion trajectory of the object is predicted according to the initial position, velocity information and motion law of the object.

[0161] Compare the first motion trajectory with the second motion trajectory to determine whether there is an intersection or overlapping area between the two. If so, further analyze the time when the two arrive at the intersection. If the time of the two is close and the difference is within a certain threshold range, it indicates that the collision risk is high; if the time difference between the two is large or the trajectories do not intersect at all, then the collision risk is low. It should be noted that the surrounding environmental factors of the vehicle, the speed change trend of the vehicle and the object, etc. can also be considered to evaluate the collision risk.

[0162] Afterwards, the vehicle image is displayed on the display screen, allowing the driver to intuitively see the visual image of the vehicle itself and the surrounding environment, and understand the current state of the vehicle and the approximate distribution of surrounding objects. At the same time, the collision risk information is conveyed to the driver through output devices (such as voice broadcast system, text prompts on the vehicle display screen, etc.).

[0163] Through the above-mentioned embodiments, by accurately predicting vehicle driving and determining collision risks and intuitively displaying them on the display screen, the driver can understand the potential dangers of road conditions in real time and clearly, make reasonable driving decisions in advance, and effectively avoid or reduce the probability of collision accidents.

[0164] Through the streaming media rearview mirror, various signals can be interacted with the information of the rearview mirror inside the car to realize a variety of assisted driving functions, such as steering and lane change collision avoidance reminder assistance, blind spot monitoring, reversing collision avoidance reminder assistance, etc. These functions can provide the driver with visual and auditory prompts, assisted braking, assisted lane keeping, etc., to improve driving safety.

[0165] In some embodiments, after analyzing the collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory, the method further includes:

[0166] Step b1, when the vehicle is in a state of turning or changing lanes, generating a torque request according to the current vehicle driving data of the vehicle and the collision risk;

[0167] Step b2, sending a torque request to a controller of the electric power steering system of the vehicle through a CAN interface, so that the electric power steering system responds to the torque request and warns the driver.

[0168] The interior rearview mirror is equipped with a CAN interface, which communicates with the vehicle's various systems / module controls and is used to send various information / signals to the electric power steering system. The electric power steering system adjusts the size and direction of the electric power assistance in real time based on this information / signal to provide appropriate steering assistance, allowing the driver to easily control the vehicle steering.

[0169] When the vehicle is in a turning and lane change operation and encounters a critical situation, such as suddenly detecting a vehicle approaching rapidly in the adjacent lane that may collide, the streaming media rearview mirror will quickly calculate and generate the corresponding torque request signal based on the preset safety strategy and sensor feedback information, and send it to the electric power steering system via the CAN bus.

[0170] According to ISO8855, this torque request signal has a clear direction definition, with a positive value indicating a torque demand to the left and a negative value indicating a torque demand to the right. After receiving the torque request, the electric power steering system immediately starts its motor control system to generate a torque output that matches the request and acts on the vehicle's steering mechanism. The application of this torque is intended to assist the driver in quickly and accurately adjusting the vehicle's driving direction to avoid collisions or other dangerous situations.

[0171] In some embodiments, the electric power steering system includes a steering wheel, the torque request includes a vibration frequency and amplitude, and the method further includes:

[0172] Step c1, sending a torque request to a controller of the electric power steering system of the vehicle through a CAN interface, so that the electric power steering system responds to the torque request and controls the steering wheel to vibrate and warn with the received vibration frequency and amplitude.

[0173] The electric power steering system also has the ability to vibrate the steering wheel according to the torque request. When the streaming rearview mirror detects a dangerous situation and determines that it is necessary to warn the driver, it sends a request signal containing specific vibration frequency and amplitude information to the electric power steering system. After receiving the signal, the electric power steering system drives its built-in vibration motor or actuator to make the steering wheel vibrate according to the specified frequency and amplitude.

[0174] As an example, different vibration frequencies and amplitudes may correspond to different degrees of danger levels or warning information, and different degrees of danger levels or warning information are determined according to any one or at least one of the following factors: collision risk, distance information, and degree of change in torque request. For example, vibrations with lower frequencies and smaller amplitudes may indicate a slight potential danger, reminding the driver to pay attention; while vibrations with higher frequencies and larger amplitudes indicate an urgent and serious danger, prompting the driver to take emergency braking or evasive measures immediately.

[0175] Through this embodiment, when the driver faces potential danger, he can quickly obtain warning information through intuitive steering wheel vibration perception without having to divert too much attention to check other devices, thereby improving the timeliness and effectiveness of the warning, thereby improving the safety of the vehicle during operations such as turning and changing lanes.

[0176] In some embodiments, after analyzing the collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory, the method further includes:

[0177] Step d1, when the vehicle is in a reversing state, generating a braking request according to the current vehicle driving data of the vehicle and the collision risk;

[0178] Step d2, sending a torque request to the braking system of the vehicle through the CAN interface, so that the braking system responds to the braking request and performs auxiliary braking on the vehicle.

[0179] When the vehicle is in reverse and encounters a critical situation, such as an object approaching quickly and very close to the rear of the vehicle, or the reversing image shows an obstacle that is about to collide with the rear, the streaming media rearview mirror will calculate the required braking request signal based on the degree of collision risk and the vehicle's driving data, and transmit it to the vehicle's braking system / brake control module through the CAN bus.

[0180] After receiving the braking request, the vehicle's braking system / brake control module will immediately start its emergency braking procedure, thereby ensuring the safety of the vehicle during emergency braking to the greatest extent possible, effectively preventing the occurrence of dangerous situations, protecting the safety of the vehicle and its occupants, and reducing potential harm to the surrounding environment and other traffic participants.

[0181] In some other embodiments, the interior rearview mirror includes a light control circuit, and the method further includes:

[0182] Step e1, when the vehicle is in a reversing state, sending a control instruction to the light control circuit of the interior rearview mirror to control at least one light area of ​​the interior rearview mirror to light up.

[0183] The interior rearview mirror includes a light control circuit, which can control the brightness of at least one light area of ​​the interior rearview mirror. Once there is a risk of collision between an object and the vehicle, a control instruction (such as a lighting instruction) is sent to the interior rearview mirror light control circuit of the vehicle to light up the interior rearview mirror light. After the interior rearview mirror light is lit, a striking visual warning signal can be formed within the driver's field of vision. Since the driver usually frequently observes the interior rearview mirror to grasp the overall situation behind when reversing, this light reminder method can directly attract the driver's attention, making him quickly aware of the dangerous situation behind, and avoiding him ignoring potential risks in other directions due to focusing on the rear. At the same time, there is no need to be distracted to check other complex dashboard warning lights or display information, thereby improving the safety of reversing operations.

[0184] The present application provides a streaming media rearview mirror and a driving assistance method thereof, which obtains a vehicle image after obtaining environmental information behind the vehicle through a camera, and clearly presents the vehicle image through a streaming media display screen, so that the driver does not need to turn his head or rely on the limited field of view of a traditional rearview mirror to obtain clear and effective environmental information behind the vehicle, which is helpful for assisted driving. In addition, the distance between objects behind the vehicle is accurately identified through a distance measurement algorithm and presented on the streaming media rearview mirror display screen or output through an output device, so that the driver can easily obtain the rear distance information without turning his head or relying on the limited field of view of a traditional rearview mirror, so that the distance between vehicles can be judged more accurately, providing assistance in driving safety, effectively reducing traffic accidents caused by blind spots in vision or misjudgment of distance, and significantly improving driving safety and convenience.

[0185] The implementation process of the functions and effects of each module / submodule / unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and the same technical effect can be achieved, so it will not be repeated here.

[0186] It should be noted that the technical solutions or technical features described in the above embodiments can be combined or supplemented with each other without causing conflicts. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the drawings; all modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. An assisted driving method based on a streaming media rearview mirror, characterized in that: The streaming media rearview mirror includes a camera and an interior rearview mirror including a display screen, and the method includes: Acquire a vehicle image obtained by collecting environmental information behind the vehicle through the camera; the environmental information includes the photographed object; Determine the distance information of the object relative to the vehicle based on a preset distance measurement algorithm; The vehicle image is displayed on the display screen and the distance information is outputted through an output device for viewing by a driver.

2. The assisted driving method based on streaming media rearview mirror according to claim 1, characterized in that: After determining the distance information of the object relative to the vehicle based on the preset distance measurement algorithm, the method further includes: Predicting a first motion trajectory of the vehicle according to first speed information of the vehicle and vehicle driving data; Determining second speed information and a second motion trajectory of the object based on the distance information and the interpolation algorithm; analyzing a collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory; The vehicle image is displayed on the display screen and the collision risk is outputted through an output device for viewing by a driver.

3. The assisted driving method based on streaming media rearview mirror as claimed in claim 2, characterized in that: After analyzing the collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory, the method further includes: When the vehicle is in a state of turning or changing lanes, generating a torque request according to current vehicle driving data of the vehicle and the collision risk; A torque request is sent to a controller of the electric power steering system of the vehicle through a CAN interface, so that the electric power steering system responds to the torque request and warns the driver.

4. The assisted driving method based on streaming media rearview mirror as claimed in claim 3, characterized in that: The electric power steering system includes a steering wheel, the torque request includes a vibration frequency and amplitude, The method further comprises: A torque request is sent to a controller of the electric power steering system of the vehicle through a CAN interface, so that the electric power steering system responds to the torque request and controls the steering wheel to vibrate and warn with the received vibration frequency and amplitude.

5. The assisted driving method based on streaming media rearview mirror as claimed in claim 2, characterized in that: After analyzing the collision risk between the object and the vehicle based on the first motion trajectory and the second motion trajectory, the method further includes: When the vehicle is in a reversing state, generating a braking request according to current vehicle driving data of the vehicle and the collision risk; A torque request is sent to the braking system of the vehicle through the CAN interface, so that the braking system responds to the braking request and performs auxiliary braking on the vehicle.

6. The assisted driving method based on streaming media rearview mirror as claimed in claim 5, characterized in that: The interior rearview mirror comprises a light control circuit, The method further comprises: When the vehicle is in a reversing state, a control instruction is sent to a light control circuit of the interior rearview mirror to control at least one light area of ​​the interior rearview mirror to light up.

7. The assisted driving method based on streaming media rearview mirror according to claim 1, characterized in that: The output device includes at least one of a display screen and a voice device.

8. The assisted driving method based on streaming media rearview mirror according to claim 7, characterized in that: The displaying the vehicle image through the display screen and outputting the distance information through an output device includes: Based on the display mode corresponding to the distance information, the vehicle image and the distance information are displayed on the display screen, and the display mode is switched according to the degree of change of the distance information.

9. The assisted driving method based on streaming media rearview mirror as claimed in claim 8, characterized in that: The display mode includes a layered display mode, The displaying the vehicle image and the distance information through the display screen includes: Acquire the vehicle image as a base layer; The distance information is superimposed on the corresponding position of the vehicle in the basic layer in the form of annotation, and the transparency of the annotated distance information decreases as the distance increases.

10. The assisted driving method based on streaming media rearview mirror according to claim 1, characterized in that: The determining the distance information of the object relative to the vehicle based on a preset distance measurement algorithm includes: Detecting the vehicle image, and determining the road plane coordinates of the object in the coordinate system of the vehicle based on the internal and external parameters of the camera; The distance information of the object relative to the vehicle is determined by a preset distance measurement algorithm and the road plane coordinates.

11. The assisted driving method based on streaming media rearview mirror according to claim 10, characterized in that: The method further comprises: Obtaining current posture information of the camera; The internal and external parameters of the camera are updated according to the posture information.

12. A streaming media rearview mirror, characterized in that: The streaming media rearview mirror includes a camera and an interior rearview mirror including a display screen; The camera includes a memory, a processor, and an assisted driving program based on a streaming media rearview mirror stored in the memory and running on the processor. When the processor executes the assisted driving program based on a streaming media rearview mirror, the steps of the assisted driving method based on a streaming media rearview mirror as described in any one of claims 1-11 are implemented.

13. A streaming media rearview mirror, characterized in that: The streaming media rearview mirror includes a camera and an interior rearview mirror including a display screen; The interior rearview mirror includes a memory, a processor, and an assisted driving program based on a streaming media rearview mirror stored in the memory and running on the processor. When the processor executes the assisted driving program based on the streaming media rearview mirror, the steps of the assisted driving method based on a streaming media rearview mirror as described in any one of claims 1-11 are implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an assisted driving program based on a streaming media rearview mirror, and when the assisted driving program based on a streaming media rearview mirror is executed, the steps of the assisted driving method based on a streaming media rearview mirror as described in any one of claims 1-11 are implemented.

Images