Driving early warning method and device, electronic equipment and storage medium
By installing a mobile terminal on the vehicle, using its computing resources to process real-time video images and adjust the position, an early warning strategy is built, which solves the problem that it is difficult to achieve when a vehicle driving warning function is not equipped with a front-view camera, and improves the accuracy and convenience of the early warning.
Patent Information
- Application Number
- CN202411085468.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-19
AI Technical Summary
In the prior art, vehicles not equipped with front-view cameras need to be equipped with additional cameras and computing equipment, resulting in high costs and complex installations, making it difficult to realize the driving warning function.
By installing mobile terminals on vehicles, using their own computing resources or networking to use cloud computing, obtain and process real-time video images, adjust the terminal position to be consistent with the vehicle coordinate system, and build early warning strategies to trigger early warning measures.
It realizes fast and convenient driving warning on vehicles without forward cameras, reduces image distortion and misjudgment, improves the accuracy and timeliness of early warnings, and avoids accidents.
Smart Images

Figure CN118977706A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving technology, and specifically to a driving warning method, device, electronic device, and computer-readable storage medium. Background Art
[0002] At present, forward-looking cameras are generally installed in newly released car models to capture the road conditions in front of the vehicle. The early warning system is mainly implemented by obtaining the lane lines and obstacles from the forward-looking camera image, and then obtaining the relative position relationship between the lane lines and obstacles and the vehicle based on the internal and external camera parameters of the camera, so as to determine whether a danger warning is needed.
[0003] However, for a large number of vehicles that are not equipped with forward-looking cameras, if they want to have this function, they need to install additional cameras and equip computing equipment to achieve it. This solution has the problems of high cost and complex installation of software and hardware equipment. Therefore, by installing a mobile terminal in a specific area, the image provided by the camera behind the mobile terminal can be used, and the computing resources of the mobile terminal can be used, or the mobile terminal can be connected to the Internet to use cloud computing, so as to conveniently and quickly implement driving warnings on vehicles without forward-looking cameras. Summary of the invention
[0004] To solve the above technical problems, the embodiments of the present application provide a driving warning method and device, an electronic device, and a computer-readable storage medium.
[0005] According to one aspect of an embodiment of the present application, a driving warning method is provided, comprising: obtaining an installation position corresponding to a mobile terminal, the mobile terminal being installed in an adjustable area of a vehicle; determining a coordinate position corresponding to the mobile terminal in the vehicle coordinate system based on a vehicle coordinate system corresponding to the vehicle, the vehicle coordinate system being used to characterize a relative position relationship between the real world and the vehicle; obtaining a real-time video image corresponding to the real world captured by the mobile terminal, and adjusting the installation position of the mobile terminal based on the real-time video image and the coordinate position so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system; constructing a warning strategy for the vehicle based on a valid image captured by the adjusted mobile terminal and performance parameters corresponding to the mobile terminal, so as to control the vehicle to execute the warning strategy when the valid image meets a warning trigger condition in the warning strategy.
[0006] According to one aspect of an embodiment of the present application, the acquiring of a real-time video image corresponding to the real world captured by the mobile terminal and adjusting the installation position of the mobile terminal based on the real-time video image and the coordinate position include: acquiring a real-time video image captured by the mobile terminal within a continuous time period; if the real-time video image is not within a valid area, calculating the attitude deflection amount of the coordinate position in a calibrated coordinate system, the origin of the calibrated coordinate system being the coordinate position corresponding to the mobile terminal in the vehicle coordinate system; and adjusting the installation position of the mobile terminal based on the attitude deflection amount.
[0007] According to one aspect of an embodiment of the present application, the method also includes: obtaining a corresponding horizontal line in the real-time video image and a standard horizon corresponding to the real-time video image; calculating a horizontal angle between the horizontal line and the standard horizon to determine a horizontal posture deflection amount of the mobile terminal in the calibrated coordinate system based on the horizontal angle; obtaining a vanishing point on the real-time video image, the vanishing point being an intersection point between parallel lines in the real-time video image; calculating a coordinate offset of the vanishing point in the calibrated coordinate system, and determining a lateral posture deflection amount and a longitudinal posture offset of the mobile terminal in the calibrated coordinate system based on the coordinate offset.
[0008] According to one aspect of an embodiment of the present application, the method also includes: generating effective areas corresponding to the vanishing points and the horizontal lines in the real-time video image based on the posture deflection amount of the mobile terminal in the calibrated coordinate system; adjusting the installation position of the mobile terminal based on the effective area so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system.
[0009] According to one aspect of an embodiment of the present application, the early warning strategy for the vehicle is constructed based on the effective image captured by the adjusted mobile terminal and the performance parameters corresponding to the mobile terminal, including: obtaining parameter information corresponding to the real-time video image, the parameter information including the height and width of the image; determining the center point and focal length corresponding to the real-time video image based on the width and height of the image; determining the internal and external parameters corresponding to the mobile terminal based on the center point and the focal length, wherein the internal and external parameters are used to establish a mapping relationship between a world coordinate system and a vehicle coordinate system; determining the distance parameter between the real-world content in the effective image based on the mapping relationship, and generating the early warning strategy for the vehicle based on the distance parameter.
[0010] According to one aspect of an embodiment of the present application, the method includes: obtaining a distortion coefficient corresponding to the mobile terminal, and performing distortion processing on the real-time video image based on the distortion coefficient to obtain an undistorted image; converting the undistorted image into an undistorted original image based on an intrinsic parameter corresponding to the mobile terminal, the intrinsic parameter including a performance parameter of the mobile terminal, wherein the intrinsic parameter is used to establish a mapping relationship between a camera coordinate system of the mobile terminal and the pixel coordinate system; converting the undistorted original image into an image in the vehicle coordinate system based on an extrinsic parameter corresponding to the mobile terminal, wherein the extrinsic parameter is used to establish a mapping relationship between the camera coordinate system and a world coordinate system; establishing a mapping relationship between the vehicle coordinate system and the world coordinate system based on the coordinate position, the mapping relationship between the camera coordinate system and the pixel coordinate system, and the mapping relationship between the camera coordinate system and the world coordinate system.
[0011] According to one aspect of an embodiment of the present application, the method also includes: performing perspective projection transformation on the camera coordinate system to obtain an image coordinate system, so as to obtain a pixel coordinate system corresponding to the real-time video image based on the image coordinate system; and generating a normalized imaging plane corresponding to the real-time image based on the property parameters corresponding to the mobile terminal and the pixel coordinate system.
[0012] According to one aspect of an embodiment of the present application, the method also includes: determining the relative rotation and translation matrix of the mobile terminal relative to the calibration coordinate system based on the intrinsic parameters of the mobile terminal, the vanishing point in the original image, and the angle of the horizontal line; obtaining the rotation and translation matrix of the mobile terminal in the vehicle coordinate system based on the coordinate position of the mobile terminal; and determining the external parameters of the mobile terminal based on the relative rotation and translation matrix and the rotation and translation matrix in the vehicle coordinate system.
[0013] According to one aspect of an embodiment of the present application, a driving warning device is provided, including: an acquisition module, used to acquire an installation position corresponding to a mobile terminal on a vehicle, the mobile terminal being installed in an adjustable area of the vehicle; a determination module, used to determine the coordinate position corresponding to the mobile terminal in the vehicle coordinate system based on a vehicle coordinate system corresponding to the vehicle, the vehicle coordinate system being used to characterize the relative position relationship between the real world and the vehicle; an adjustment module, used to acquire a real-time video image corresponding to the real world captured by the mobile terminal, and adjust the installation position of the mobile terminal based on the real-time video image and the coordinate position so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system; a construction module, used to construct a warning strategy for the vehicle based on the effective image captured by the adjusted mobile terminal and the performance parameters corresponding to the mobile terminal, so as to control the vehicle to execute the warning strategy when the effective image meets the warning trigger condition in the warning strategy.
[0014] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the traffic warning method as described above.
[0015] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the driving warning method as described above.
[0016] According to one aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which implements the steps in the driving warning method as described above when executed by a processor.
[0017] In the technical solution provided in the embodiments of the present application, by installing the mobile terminal on the vehicle, the vehicle without its own image acquisition device can collect external image information, and the coordinate position of the mobile terminal in the vehicle coordinate system is determined by the vehicle coordinate system corresponding to the vehicle, and then by obtaining the real-time video image corresponding to the real world captured by the mobile terminal, the installation position of the mobile terminal is adjusted based on the real-time video image and the coordinate position, so that the posture of the mobile terminal is consistent with the vehicle coordinate system, which can ensure that the collected video image more accurately reflects the actual environment around the vehicle, and reduce misjudgment caused by image distortion or dislocation. Then, according to the adjusted effective image and the vehicle's early warning strategy, potential road hazards can be identified more promptly and corresponding early warning measures can be triggered, thereby helping the driver to react in advance and avoid accidents.
[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0020] Figure 1 is a schematic diagram of an implementation environment for driving warning shown in an exemplary embodiment of the present application;
[0021] Figure 2is a flow chart of a driving warning method shown in an exemplary embodiment of the present application;
[0022] Figure 3 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0023] Figure 4 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0024] Figure 5 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0025] Figure 6 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0026] Figure 7 is a schematic diagram of an effective area of a mobile terminal for driving warning shown in an exemplary embodiment;
[0027] Figure 8 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0028] Fig. 9 is a schematic diagram of a driving scene of a driving warning shown by an exemplary embodiment;
[0029] Fig.10 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0030] Fig.11 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0031] Fig.12 is a schematic diagram of imaging from an original image to the real world, shown in an exemplary embodiment;
[0032] Fig.13 is a flow chart of a driving warning method shown in another exemplary embodiment of the present application;
[0033] Fig.14 It is a brief flow chart of driving warning in an exemplary application scenario;
[0034] Fig.15 is a block diagram of a traffic warning device shown in an exemplary embodiment of the present application;
[0035] Fig.16 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION
[0036] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0038] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0039] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0040] First of all, it should be noted that navigation is a technology that points to the critical path of the destination. It is the process of monitoring and controlling the movement of objects such as processes, vehicles, and pedestrians from one location to another. In a broader sense, navigation can refer to any skill or study involving determining position and direction. The field of navigation is generally divided into four areas: land navigation, marine navigation, aviation navigation, and space navigation. The embodiments of this application involve land navigation, which is used to monitor the position change process of objects such as vehicles and pedestrians moving from one location to another along a planned path.
[0041] During the navigation process, the road condition information will be refreshed at a specific frequency to ensure user experience. Taking the application scenario of vehicle route navigation as an example, due to the high speed of the vehicle, the road condition of the vehicle's driving road will be refreshed at the minute level during the navigation process. For example, the remaining time for the vehicle to reach the destination is usually displayed in minutes as the minimum unit. Therefore, the road condition of the vehicle's driving road will be refreshed every 1 minute during the navigation process, so that the latest road condition can be synchronized to the driver in time. In other application scenarios, the frequency of refreshing the road condition of the object's driving road can be set according to actual conditions, and the embodiments of the present application are not limited to this.
[0042] Driving warning systems do play a vital role in public transportation safety, especially in preventing and reducing accidents caused by improper driver operation or fatigue driving. These systems can monitor the environment around the vehicle and the driver's status in real time by integrating multiple sensors, cameras, radars and advanced algorithms, thereby providing timely warnings and auxiliary control, significantly improving the safety of road driving. Among them, driving warning systems include lane departure warnings and collision warnings. The lane departure warning system monitors the relative position of the vehicle and the lane markings through cameras or radars. When the system determines that the vehicle may unconsciously deviate from the current lane, it will warn the driver, usually through sound, vision or touch (such as steering wheel vibration). This is especially important for drivers whose attention has decreased after long-term driving, and can effectively avoid lane departure accidents caused by fatigue or distraction; the collision warning system monitors the road conditions ahead through forward radars or cameras, and warns the driver in advance when it detects the risk of collision with the vehicle ahead, pedestrians or other obstacles. Some systems can also provide different levels of warnings according to the degree of danger, and automatically take braking measures in an emergency (AEB, Autonomous Emergency Braking) to further reduce the risk and severity of collisions.
[0043] Figure 1 FIG. 1 is a schematic diagram of an implementation environment for displaying a driving warning during navigation according to an exemplary embodiment of the present application. Figure 1As shown, during the driving process of the vehicle, navigation is achieved through the navigation map software installed on the mobile terminal 110, and the mobile terminal 110 is installed in the adjustable area of the vehicle, and then the server 120 obtains the vehicle coordinate system corresponding to the vehicle, and determines the coordinate position corresponding to the mobile terminal in the vehicle coordinate system, wherein the vehicle coordinate system is used to characterize the relative position relationship between the real world and the vehicle, and then the server 120 obtains the real-time video image corresponding to the real world captured by the mobile terminal 110, and then adjusts the installation position of the mobile terminal based on the real-time video image and the coordinate position of the mobile terminal 110, so that the posture of the mobile terminal 110 is consistent with that of the vehicle coordinate system, and then the vehicle driving warning system is constructed based on the effective image captured by the adjusted mobile terminal 110 and the performance parameters corresponding to the mobile terminal 110, so as to control the vehicle to execute the warning strategy when it is determined based on the effective image in the mobile terminal 110 that the warning trigger condition in the trigger warning strategy is currently met, wherein the warning strategy includes lane departure warning and collision warning.
[0044] in, Figure 1 The mobile terminal 110 shown can be any terminal device that supports installation of navigation map software and image acquisition, such as a smart phone, a car computer, a tablet computer, a laptop computer, or a wearable device, but is not limited thereto. Figure 1 The server 120 shown can be, for example, an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, and is not limited here. The mobile terminal 110 can communicate with the server 120 through wireless networks such as 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), and 5G (fifth generation mobile information technology), and is not limited here.
[0045] For a large number of vehicles that are not equipped with forward-looking cameras at present, if they want to have this function, they need to install additional cameras and equip computing equipment to realize it. This solution has the problems of high cost and complex installation of software and hardware equipment. Therefore, by installing a mobile terminal in a specific area, the image provided by the camera behind the mobile terminal can be used, and the computing resources of the mobile terminal can be used, or the mobile terminal can be connected to the Internet to use cloud computing, so as to conveniently and quickly implement driving warning on vehicles without forward-looking cameras.
[0046] The above-mentioned problems are generally applicable in common travel scenarios. To solve these problems, the embodiments of the present application respectively propose a driving warning method, a driving warning device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail below.
[0047] See also Figure 2 , Figure 2 is a flow chart of a driving warning method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is specifically executed by the server 120 in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.
[0048] like Figure 2 As shown, in an exemplary embodiment, the specific implementation process of the driving warning method includes at least steps S210 to S240, which are described in detail as follows:
[0049] Step S210, obtaining an installation position corresponding to the mobile terminal, where the mobile terminal is installed in an adjustable area of the vehicle.
[0050] Specifically, for vehicles that are not equipped with a forward-looking image acquisition device, if a driving warning function is required, an additional camera needs to be installed. Therefore, a mobile terminal with an image acquisition function can be installed in an adjustable area of the vehicle, and then the environmental image around the vehicle can be collected through the mobile terminal. By using the computing resources of the mobile terminal itself or the mobile terminal being connected to the Internet and using cloud computing, driving warnings can be easily and quickly implemented on vehicles without forward-looking cameras.
[0051] Exemplarily, the mobile terminal can be installed around the dashboard of the vehicle, so that the driver can view the information on the mobile terminal during driving, such as navigation, music playback, etc. However, it should be noted that the installation position will not block the driver's line of sight and avoid affecting the deployment of the vehicle's airbags. In some exemplary embodiments, the mobile terminal can also be installed on the central console of the vehicle to facilitate the driver to operate or use the mobile terminal.
[0052] Step S220, determining the coordinate position of the mobile terminal in the vehicle coordinate system based on the vehicle coordinate system corresponding to the vehicle, where the vehicle coordinate system is used to represent the relative position relationship between the real world and the vehicle.
[0053] It should be noted that the vehicle coordinate system is a special moving coordinate system used to describe the movement of a car. Its characteristics include: the origin coincides with the center of mass of the vehicle. When the vehicle is stationary on a horizontal road, the X-axis is parallel to the ground and points to the front of the vehicle. The Z-axis passes through the center of mass of the car and points upward. The Y-axis points to the driver's left (or right, depending on the definition, but usually the left).
[0054] Specifically, the origin of the vehicle coordinate system can be determined, and the center of mass of the vehicle can be used as the origin of the vehicle coordinate system. Then, the coordinate position of the mobile terminal in the vehicle coordinate system can be determined by measuring the relative distance and direction between the mobile terminal and a known point in the vehicle (such as a mark on the dashboard, a position on the seat, etc.). Of course, in some feasible embodiments, the position of the mobile terminal can be sensed by on-board sensors (such as cameras, radars, etc.), thereby determining the corresponding coordinate position of the mobile terminal in the vehicle coordinate system.
[0055] Step S230, obtaining a real-time video image corresponding to the real world captured by the mobile terminal, and adjusting the installation position of the mobile terminal based on the real-time video image and the coordinate position so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system.
[0056] Specifically, first, ensure that the mobile terminal (such as a smart phone, tablet computer or vehicle camera) has established a connection with the vehicle system (such as a vehicle computer or display screen) and is able to transmit video images in real time. Then, through the camera function of the mobile terminal, real-time video images of the real world are collected. The real-time video image is preliminarily processed, and the preliminary processing may include denoising, contrast enhancement, edge detection, etc., so as to more accurately identify key features in the image. Then, the real-time video image after preliminary processing can be subjected to posture recognition, wherein posture recognition includes evaluating the current posture of the mobile terminal (i.e., its direction and angle relative to the vehicle coordinate system) through video image analysis. This can be done by detecting the gravity vector, horizon or other reference lines in the image. Then, the posture recognition result is compared with the ideal posture of the vehicle coordinate system. The ideal posture means that the camera lens of the mobile terminal is completely parallel to the front of the vehicle coordinate system (i.e., the X-axis direction), and the vertical direction (Y-axis and Z-axis) also conforms to the definition of the vehicle coordinate system.
[0057] Afterwards, based on the results of the posture evaluation, corresponding adjustment instructions are generated. These instructions will instruct how to move or rotate the mobile terminal to achieve a posture consistent with the vehicle coordinate system. The installation position of the mobile terminal can be fine-tuned according to the adjustment instructions through the vehicle system or manually, which may involve rotating the camera module, adjusting the angle of the bracket, or repositioning the entire mobile terminal.
[0058] In addition, during the adjustment process, real-time video images can be continuously acquired and analyzed to verify the adjustment effect. When the adjustment is completed, it is confirmed again whether the posture of the mobile terminal is completely consistent with the vehicle coordinate system. This can be done through visual inspection or further image processing algorithms.
[0059] Step S240, constructing a warning strategy for the vehicle based on the adjusted valid image collected by the mobile terminal and the performance parameters corresponding to the mobile terminal, so as to control the vehicle to execute the warning strategy when the valid image meets the warning triggering condition in the warning strategy.
[0060] Specifically, the real-time video images collected from the mobile terminal are preprocessed, including denoising, contrast enhancement, edge detection, etc., to improve image quality and highlight key information, and key features such as lane lines, traffic signs, vehicles ahead, pedestrians, etc. are extracted from the preprocessed images. These features will be used for subsequent warning judgments. Consider the performance parameters of the mobile terminal's camera resolution, frame rate, field of view, image processing capabilities, etc. These parameters will affect the quality and real-time performance of the image, and thus affect the accuracy and reliability of the warning strategy. At the same time, the vehicle's speed, acceleration, steering angle and other state information are considered to more accurately judge the vehicle's driving status and potential risks. Based on the feature information and vehicle status in the effective image, a series of warning trigger conditions are defined. For example, lane departure, sudden deceleration of the vehicle ahead, pedestrians crossing the road, etc., and corresponding warning measures are formulated for each warning trigger condition. These measures may include sound warnings, visual prompts (such as warning lights on the dashboard), vibration feedback, and automatic avoidance measures (such as emergency braking, steering assistance, etc.).
[0061] In addition, the adjusted mobile terminal can be used to continuously collect effective images and analyze the feature information and vehicle status in the image in real time. When the effective image meets a warning trigger condition in the driving warning strategy, the warning judgment is immediately made. In addition, feedback from drivers and passengers on the driving warning strategy is collected to understand their feelings about the warning effect and comfort. The data generated during the warning process is deeply analyzed to evaluate the accuracy and effectiveness of the warning strategy. Based on user feedback and data analysis results, the driving warning strategy is continuously optimized to improve its adaptability and practicality.
[0062] In this embodiment, by installing the mobile terminal on the vehicle, the vehicle without its own image acquisition device can collect external image information, and the coordinate position of the mobile terminal in the vehicle coordinate system is determined by the vehicle coordinate system corresponding to the vehicle, and then the real-time video image corresponding to the real world acquired by the mobile terminal is obtained, and the installation position of the mobile terminal is adjusted based on the real-time video image and the coordinate position, so that the posture of the mobile terminal is consistent with the vehicle coordinate system, which can ensure that the collected video image more accurately reflects the actual environment around the vehicle and reduce misjudgment caused by image distortion or dislocation. Then, based on the adjusted effective image and the vehicle's early warning strategy, the system can identify potential road hazards (such as obstacles ahead, pedestrians crossing, etc.) more timely, and trigger corresponding early warning measures, such as sounding an alarm, vibrating the seat, etc., so as to help the driver react in advance and avoid accidents.
[0063] Further, based on the above embodiments, please refer to Figure 3 In one of the exemplary embodiments provided in the present application, the specific implementation process of acquiring the real-time video image corresponding to the real world captured by the mobile terminal and adjusting the installation position of the mobile terminal based on the real-time video image and the coordinate position includes steps S310 to S330, which are described in detail as follows:
[0064] Step S310, acquiring real-time video images collected by the mobile terminal in a continuous time period.
[0065] Specifically, ensure that the mobile terminal is powered on and connected to the vehicle system (if applicable), and check and adjust the camera settings of the mobile terminal, including resolution, frame rate, exposure, etc., to ensure that the quality and real-time performance of the video image meet the requirements. Configure the video acquisition parameters in the application, such as video source (usually camera), video format, encoding method, etc., and ensure that the application has sufficient permissions to access the camera and store video data.
[0066] Step S320, if the real-time video image is not within the effective area, the attitude deflection of the coordinate position in the calibration coordinate system is calculated, and the origin of the calibration coordinate system is the coordinate position corresponding to the mobile terminal in the vehicle coordinate system.
[0067] Specifically, the vehicle coordinate system is usually a three-dimensional coordinate system with a fixed point of the vehicle (such as the center of mass of the vehicle, the center of the rear axle, etc.) as the origin, and has the directions of forward (X-axis), lateral (Y-axis) and vertical upward (Z-axis). The calibration coordinate system is established based on the vehicle coordinate system, and its origin is located at the coordinate position of the mobile terminal (camera) in the vehicle coordinate system. This means that the calibration coordinate system will change with the installation position of the camera on the vehicle. Through image processing technology (such as image recognition, feature point detection, etc.), the position of a key point (such as the center of the image, the center of a specific object, etc.) in the real-time video image in the image coordinate system can be determined. Then, using the internal and external parameters of the camera (obtained through calibration), this two-dimensional image coordinate can be converted into a three-dimensional world coordinate (i.e., the coordinate in the vehicle coordinate system). According to the converted three-dimensional coordinates, it can be determined whether the point is located in a predetermined valid area. The valid area can be set based on factors such as vehicle driving safety and field of view.
[0068] Then, if the key point is not in the valid area, it is necessary to calculate the attitude deflection of the point in the calibration coordinate system. This usually involves calculating the vector between the point and the origin of the calibration coordinate system, and extracting the deflection angle (such as pitch angle, yaw angle, roll angle, etc.) from this vector. These angles describe the change in direction of the key point relative to the origin of the calibration coordinate system. Pitch angle (Pitch): rotation around the X-axis, indicating up and down movement; Yaw angle (Yaw): rotation around the Y-axis, indicating left and right movement; Roll angle (Roll): rotation around the Z-axis, indicating rotation and tilt.
[0069] Step S330: adjusting the installation position of the mobile terminal based on the posture deflection amount.
[0070] First, you need to carefully analyze the calculated attitude deflection, including pitch, yaw, and roll angles. These angles reflect the difference between the current attitude of the mobile terminal and the ideal attitude. According to the sign and size of the deflection, determine the direction and degree of adjustment. For example, if the pitch angle is positive, it means that the camera is tilted upward and its installation position needs to be adjusted downward; if the yaw angle is negative, it means that the camera is tilted to the left and needs to be adjusted to the right. After the adjustment is completed, recalculate the attitude deflection of the mobile terminal to verify whether the adjustment effect is as expected. Check the quality of the real-time video image to ensure that the image is clear, stable, and covers the predetermined effective area. If the adjustment result is not completely satisfactory, you can make fine adjustments as needed until the best effect is achieved.
[0071] In this embodiment, by adjusting the installation position of the mobile terminal, the effectiveness of the video images collected by the mobile terminal can be improved, thereby improving the accuracy of the driving warning, and further improving the safety and stability during driving.
[0072] Further, based on the above embodiments, please refer to Figure 4 In one of the exemplary embodiments provided in the present application, the specific implementation process of the above-mentioned driving warning method may also include step S410 and step S420, which are described in detail as follows:
[0073] Step S410, obtaining a corresponding horizontal line in the real-time video image and a standard horizon corresponding to the real-time video image;
[0074] Step S420, calculating the horizontal angle between the horizontal line and the standard horizon, so as to determine the horizontal posture deflection amount of the mobile terminal in the calibration coordinate system based on the horizontal angle.
[0075] Exemplarily, it is necessary to identify or extract the corresponding horizontal line from the real-time video image. This can usually be achieved through image processing techniques, such as edge detection, Hough transform and other methods to detect straight lines in the image, and select nearly horizontal straight lines as horizontal lines. Among them, the standard horizon is an imaginary reference line that should be presented as horizontal in the image. Ideally, if the mobile terminal is placed completely horizontally and the camera is facing forward, the horizon in the image (such as distant hills, sea level, etc.) should coincide with this standard horizon. However, in practical applications, it may be necessary to preset a fixed horizontal line as a standard, or dynamically determine the position of the standard horizon in some way (such as GPS and map data). Once the horizontal line and the standard horizon in the real-time video image are available, the next step is to calculate the horizontal angle between the two lines. This can be achieved by calculating the slopes (or direction vectors) of the two lines, and then using inverse trigonometric functions (such as arctan) to calculate the angle between them. Finally, based on the calculated horizontal angle θ h , the horizontal attitude deflection of the mobile terminal in the calibration coordinate system can be determined. This deflection represents the rotation angle of the mobile terminal relative to the completely horizontal state (i.e., the standard horizon). If the angle is positive, it means that the mobile terminal is rotating to the right; if the angle is negative, it means that it is rotating to the left.
[0076] In this embodiment, in real-time video image processing, the horizontal line in the video image is obtained and compared with the standard horizon, and then the horizontal angle between the two is calculated to determine the horizontal posture deflection of the mobile terminal in the calibration coordinate system, improve the posture measurement accuracy, reduce error accumulation, and improve user experience.
[0077] Further, based on the above embodiments, please refer to Figure 5 In one of the exemplary embodiments provided in the present application, the specific implementation process of the above-mentioned driving warning method may also include step S510 and step S520, which are described in detail as follows:
[0078] Step S510, obtaining a vanishing point on the real-time video image, where the vanishing point is an intersection point between parallel lines in the real-time video image;
[0079] Step S520, calculating the coordinate offset of the vanishing point in the calibration coordinate system, and determining the lateral posture deflection and the longitudinal posture offset of the mobile terminal in the calibration coordinate system based on the coordinate offset.
[0080] It should be noted that the vanishing point (VP) is the point where parallel lines in the image intersect in the distance. In perspective projection, these points are very useful for understanding the spatial layout in the image and calculating the camera pose.
[0081] Specifically, the edges in the image can be identified by an edge detection algorithm (such as the Canny edge detector), and then the Hough Transform or a similar algorithm is used to detect the straight lines in the image. Through the detected straight lines, the intersections between them are found, and these intersections are the vanishing points. Then, the vanishing points can be calculated, and then the detected vanishing points can be transformed from the image coordinate system to the calibration coordinate system. This usually involves the transformation of the image to the world coordinate system, including the intrinsic parameters (focal length, optical center, etc.) and extrinsic parameters (rotation, translation, etc.) of the camera.
[0082] Then, according to the position of the vanishing points in the calibration coordinate system, their deviation from the standard position (such as the position of the vanishing point when facing forward) is analyzed. The deflection and offset are calculated, where the lateral attitude deflection is usually related to the lateral offset of the horizontal vanishing point relative to the standard position, and the longitudinal attitude offset is usually related to the longitudinal offset of the vertical vanishing point relative to the standard position. These calculations may involve geometric analysis and triangulation, depending on the definition of the calibration coordinate system and the specific parameters of the camera.
[0083] In this embodiment, the installation position of the mobile terminal is adjusted in terms of lateral posture deflection and longitudinal posture deflection, thereby improving the accuracy of mobile terminal posture assessment and enhancing user experience, so that driving warnings based on images captured by the mobile terminal are more accurate and effective.
[0084] Further, based on the above embodiments, please refer to Figure 6 In one of the exemplary embodiments provided in the present application, the specific implementation process of the above-mentioned driving warning method may also include step S610 and step S620, which are described in detail as follows:
[0085] Step S610, generating effective areas corresponding to the vanishing points and the horizontal lines in the real-time video image based on the posture deflection amount of the mobile terminal in the calibration coordinate system;
[0086] Step S620: adjusting the installation position of the mobile terminal based on the effective area so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system.
[0087] Specifically, Figure 7 As shown, based on the posture deflection of the mobile terminal, the ideal positions where the vanishing point and the horizontal line should be located in the video image can be calculated if the posture of the mobile terminal is completely consistent with the vehicle coordinate system. These ideal positions define one or more "valid areas", that is, the areas where the vanishing point and the horizontal line should be within the allowable error range.
[0088] The generation of effective areas may involve complex geometric calculations and image analysis to ensure that they are both rigorous and practical. These areas may be presented in the form of rectangles, circles, or other shapes, depending on the application scenario and algorithm design. Next, the installation position of the mobile terminal is adjusted according to the position and size of the effective area. This usually includes rotating, tilting, or translating the mobile terminal so that the vanishing point and horizontal line in the video image it captures are as close as possible to or completely fall within the effective area.
[0089] This adjustment process may require repeated iterations, because each adjustment step may affect the overall posture and shooting effect of the mobile terminal. Therefore, in actual operation, some form of feedback mechanism (such as real-time video preview, posture indicator, etc.) is usually used to help users or automatic systems make precise adjustments.
[0090] In this embodiment, by adjusting the posture of the mobile terminal and the vehicle coordinate system to be consistent, the quality of the video image collected by the mobile terminal can be improved, and the stability and safety of the driving warning can be improved.
[0091] Further, based on the above embodiment, please refer to Figure 8 In one of the exemplary embodiments provided in the present application, the specific implementation process of constructing the vehicle warning strategy based on the adjusted valid image collected by the mobile terminal and the performance parameters corresponding to the mobile terminal may also include steps S810 to S840, which are described in detail as follows:
[0092] Step S810, obtaining parameter information corresponding to the real-time video image, the parameter information including the height and width of the image;
[0093] Step S820, determining the center point and focal length corresponding to the real-time video image based on the width and height of the image;
[0094] Step S830, determining internal and external parameters corresponding to the mobile terminal based on the center point and the focal length, wherein the internal and external parameters are used to establish a mapping relationship between the world coordinate system and the vehicle coordinate system;
[0095] Step S840, determining a distance parameter between real-world contents in the effective image based on the mapping relationship, and generating a warning strategy for the vehicle based on the distance parameter.
[0096] The real-time video image is obtained through the mobile terminal. Then, the metadata of the video image is parsed or queried to obtain the basic parameter information of the image, including the height (Height) and width (Width) of the image. The center point (CenterPoint) of the image can be determined by half of the width and height of the image, that is, the coordinates of the center point are (Width / 2, Height / 2).
[0097] It should be noted that the focal length may be a virtual or assumed value here, because the on-board camera usually does not directly provide the exact value of the focal length. The focal length can be estimated based on the intrinsic properties of the camera (such as lens type, viewing angle, etc.) and the image size. The focal length is very important for subsequent 3D reconstruction and distance estimation. Intrinsic parameters mainly include focal length, optical center (i.e. center point), image center, distortion coefficient, etc., which describe the optical properties and imaging model inside the camera. Extrinsic parameters describe the position and posture of the camera in the world coordinate system, including rotation matrix and translation vector, which are used to transform points in the camera coordinate system into the world coordinate system. Through the combination of internal and external parameters, a mapping relationship from the image coordinate system to the world coordinate system can be established, which is the basis for 3D reconstruction and distance measurement. Using the established mapping relationship, the pixel points in the image can be inversely projected and converted back to points in the 3D world coordinate system. Then, by comparing the positions of different points in three-dimensional space, the actual distance between them can be calculated. These distance parameters are crucial for evaluating the safety of the vehicle's surroundings, such as detecting the distance of pedestrians, vehicles or other obstacles. Based on the calculated distance parameters, combined with dynamic information such as the vehicle's speed, acceleration, and driving direction, the safety of the current driving state can be evaluated. If a potential dangerous situation is detected (such as being too close to the vehicle in front, pedestrians crossing the road, etc.), a corresponding warning signal is generated or automatic obstacle avoidance measures are taken to improve driving safety.
[0098] Optionally, in some feasible embodiments, a certain image recognition block is responsible for perception of the forward-looking environment such as lane line recognition, pedestrian / vehicle recognition and tracking, traffic sign recognition, and traffic light recognition based on images; the coordinate conversion module is responsible for converting the recognition results from image coordinates to vehicle coordinates based on internal and external parameters of the camera, so as to obtain the distance between the vehicle and the ground recognition target; the warning trigger module performs logical judgment on the environmental recognition results established in the vehicle coordinate system, and triggers a warning when the warning conditions are met, and issues a warning to the driver.
[0099] For image recognition in the application, including but not limited to lane recognition, pedestrian / vehicle recognition and tracking, traffic sign recognition, and traffic light recognition, the specific implementation design of its functional modules requires the application of semantic segmentation, target detection, key point recognition, and text recognition based on neural networks. There are many open source implementation methods for specific models, which are not described here. The recognition module can be run on the mobile terminal or on the cloud server after being called and sent to the mobile terminal under network conditions. The specific choice should be based on the hardware configuration of the mobile terminal and the network conditions, and the recognition module with the shortest running time shall prevail.
[0100] The purpose of coordinate transformation is to establish the association between points on the image coordinates and points on the vehicle coordinates, that is, the image in pixels can be projected to the real world coordinates in distance (cm) through the internal and external parameters. For example, the lane lines on both sides of the vehicle in the middle and the vehicle in front are identified in the image, and the pixel area coordinates corresponding to the lane lines and the vehicle in front are obtained. They can be projected to the vehicle coordinates through the internal and external parameters, as shown in the figure. Fig. 9 As shown below. Taking the identification of the front vehicle and the lane lines on both sides as an example, through the projection of internal and external parameters, we can obtain the distance between the origin of the vehicle coordinate system (the projection point of the center point of the front bumper on the ground) and the rear of the obstacle vehicle in front, as well as the distance between the bottom point of the left and right lane lines and the origin of the vehicle coordinate system, which are expressed as L1 and L2 respectively.
[0101] For the warning trigger module in, based on the environmental perception data constructed in the vehicle coordinate system, it can realize including but not limited to lane departure warning, collision warning, speeding warning, etc. Based on the data listed in, collision warning and lane departure warning can be realized. For the collision warning in, the brief implementation process is as follows, where the input is the distance L2 between the obstacle vehicle and the self-vehicle and the real-time relative distance V between the self-vehicle and the obstacle vehicle. By converting, the collision value D is obtained, and according to the collision conversion factor C, it is used to better quantify the collision risk between the self-vehicle and the obstacle vehicle. When the collision value D is higher than the preset value D′, the collision warning is triggered.
[0102] The brief implementation process of lane departure warning is as follows, where the input is the distance L1 and L2 between the vehicle and the lane lines on both sides and the real-time speed u of the vehicle. The lane line departure warning conversion factor is used to better quantify the degree of lane deviation of the vehicle. When the calculated result L is higher than the threshold L', it is judged as having a risk of deviation. The type of deviation can be further judged according to the positive and negative conditions of L. When it is positive, it triggers the right deviation warning, otherwise it triggers the left deviation warning. Note that the logic here is a simple implementation form. If you need to obtain better results, you can further calculate the relationship between the trajectory of the lane line and the heading of the vehicle, and make a comprehensive judgment based on the degree of centering of the vehicle in the lane.
[0103] In this embodiment, by establishing a mapping relationship between the world coordinate system and the vehicle coordinate system, the distance parameters between the real world content in the effective image can be accurately measured. This is crucial for applications such as obstacle detection, lane keeping, and obstacle avoidance during vehicle driving. Accurate acquisition of internal and external parameters helps to determine the precise position and direction of a mobile terminal (such as a vehicle-mounted camera) in the vehicle coordinate system, thereby improving the positioning accuracy of the entire system.
[0104] Further, based on the above embodiments, please refer to Fig.10 In one of the exemplary embodiments provided in the present application, the specific implementation process of the above-mentioned driving warning method may also include steps S1010 to S1040, which are described in detail as follows:
[0105] Step S1010, obtaining a distortion coefficient corresponding to the mobile terminal, and performing distortion processing on the real-time video image based on the distortion coefficient to obtain a distortion-free image;
[0106] Step S1020: converting the undistorted image into an undistorted original image based on intrinsic parameters corresponding to the mobile terminal, wherein the intrinsic parameters include performance parameters of the mobile terminal, wherein the intrinsic parameters are used to establish a mapping relationship between a camera coordinate system and a pixel coordinate system of the mobile terminal.
[0107] Specifically, the intrinsic parameters usually include the following key elements: Focal length: The focal length describes the distance from the camera lens to the image plane (an equivalent value in optical terms, because the actual sensor may not be located at the physical focus of the lens). The focal length determines the degree of magnification of objects in the image. Optical center: The optical center is the projection of the origin of the camera coordinate system on the image plane. It is usually the center point of the image, but it may also be slightly offset due to manufacturing or installation errors. Image center: Although sometimes confused with the optical center, the image center more specifically refers to the geometric center point of the image plane, which may not completely coincide with the optical center due to the rectangular arrangement of pixels. However, in many cases, in order to simplify calculations, the optical center and the image center can be regarded as the same point. Principal point: In stricter terms, the principal point refers to the intersection of the camera optical axis and the image plane, which is usually very close to or coincident with the optical center and the image center. Pixel size: Although not always directly part of the intrinsic parameters, the pixel size (i.e., the pixel size of each pixel on the sensor) The physical size of a pixel) is critical to understanding the spatial resolution in an image. It helps convert pixel distances in an image to distances in the real world. Distortion coefficients: While the distortion coefficients themselves are not part of the intrinsic parameters, they are used along with the intrinsic parameters to correct for distortions in an image. However, when talking about camera models, intrinsic parameters and distortion coefficients are usually mentioned together because together they define how the camera maps the 3D world onto a 2D image. Determination of intrinsic parameters is typically achieved through a camera calibration process. During calibration, multiple images are taken using a calibration plate with known geometric properties (such as a checkerboard pattern) and the intrinsic parameters and (optionally) distortion coefficients are calculated from these images.
[0108] That is, intrinsic parameters play a vital role in computer vision and image processing. They are mainly used to establish the mapping relationship between the camera coordinate system of the camera (or the camera in the mobile terminal) and the pixel coordinate system. This mapping relationship allows the pixel points on the image plane to be converted back to points in the three-dimensional space, or the points in the three-dimensional space to be projected onto the image plane.
[0109] Step S1030, converting the undistorted original image into an image in the vehicle coordinate system based on the extrinsic parameters corresponding to the mobile terminal, wherein the extrinsic parameters are used to establish a mapping relationship between the camera coordinate system and the world coordinate system;
[0110] Step S1040 , establishing a mapping relationship between the vehicle coordinate system and the world coordinate system based on the coordinate position, the mapping relationship between the camera coordinate system and the pixel coordinate system, and the mapping relationship between the camera coordinate system and the world coordinate system.
[0111] The external parameters usually include the rotation matrix and the translation vector. The rotation matrix describes the direction of the camera coordinate system relative to the world coordinate system, while the translation vector describes the position of the camera in the world coordinate system. These two parameters together define the rigid body transformation from the camera coordinate system to the world coordinate system.
[0112] Using the external parameters, the points in the undistorted original image (i.e. the image that has been corrected for distortion) can be transformed from the camera coordinate system to the world coordinate system. This is usually achieved by first transforming the pixels in the image back to three-dimensional points in the camera coordinate system (which requires intrinsic parameters and depth information), and then using the external parameters to transform these points to the world coordinate system.
[0113] The vehicle coordinate system is a coordinate system with the vehicle as reference. It usually has an origin (such as the center of the vehicle or the center of the rear axle), and associated X, Y, and Z axes (which may correspond to the front and back, left and right, and up and down directions of the vehicle, respectively).
[0114] To establish the mapping relationship between the vehicle coordinate system and the world coordinate system, you need to know the position and attitude of the vehicle in the world coordinate system. This can usually be obtained through sensors on the vehicle (such as GPS, IMU, etc.). Once the position and attitude of the vehicle are known, the mapping relationship between the vehicle coordinate system and the world coordinate system can be established through rigid body transformation (similar to the transformation from the camera coordinate system to the world coordinate system).
[0115] Based on the above mapping relationship, a complete mapping chain from the vehicle coordinate system to the world coordinate system can be established. This mapping chain includes the transformation from the vehicle coordinate system to the camera coordinate system (which depends on the installation position and posture of the camera on the vehicle), the transformation from the camera coordinate system to the pixel coordinate system (defined by the intrinsic parameters), the transformation from the camera coordinate system to the world coordinate system (defined by the extrinsic parameters), and (if necessary) the transformation from the world coordinate system to another specific world coordinate system (such as another vehicle or a fixed reference point).
[0116] However, in this scenario, if the goal is to directly establish a mapping relationship between the vehicle coordinate system and the world coordinate system, and the mapping relationship between the camera coordinate system and these two coordinate systems is known, it can be achieved by combining these mapping relationships. Specifically, the points in the vehicle coordinate system can be first transformed into the camera coordinate system (which may require the position and attitude information of the vehicle relative to the camera), and then these points can be transformed into the world coordinate system using external parameters.
[0117] In addition, in order to obtain a better picture presentation and thus realize effective recognition processing, the camera is installed in the recommended area and the position of the camera relative to the vehicle coordinate system (x car ,ycar , z car ), the real-time mobile terminal rear camera image can be obtained through the mobile terminal rear camera. Based on the video data, the validity of the mobile terminal posture can be detected. The detection content mainly includes the vanishing point (X v , Y v ) and horizon detection, which can be implemented based on key point and line detection of neural network. The specific scheme is not described here. The image size of the obtained backward camera is known, with width W and height H. For the internal parameter K of the backward camera, it mainly includes the candle core point (c x , c y ) and focal length (f x , f y ), the candle point is usually the center of the image (c x =W / 2,c y =H / 2), (f x , f y ) are the focal lengths along the width and height of the image, respectively. The value in the intrinsic parameter K can be obtained by obtaining the model of the mobile phone, and the distortion coefficient D of the image can also be obtained by obtaining the model of the mobile phone. In the following text, the intrinsic parameter K and the distortion coefficient D are collectively referred to as the camera intrinsic parameter. The internal parameter establishes the mapping relationship between the camera normalized imaging plane projected from the external world and the actual image. The external parameter RT establishes the mapping relationship from the camera normalized imaging plane to the real world coordinate system. Among them, the expressions of the intrinsic parameter K and the distortion coefficient D are as follows:
[0118]
[0119] D=[k1,k2,k3,k4]
[0120] In this embodiment, by obtaining the distortion coefficient corresponding to the mobile terminal, the real-time video image is distorted to obtain a distortion-free image. This eliminates the image distortion and deformation caused by lens distortion and improves the quality and accuracy of the image. The intrinsic parameters establish the mapping relationship between the mobile terminal camera coordinate system and the pixel coordinate system, while the extrinsic parameters establish the mapping relationship between the camera coordinate system and the world coordinate system. The combination of these two mapping relationships enables each pixel in the image to accurately correspond to a position in the world coordinate system, thereby improving the accuracy of positioning and measurement.
[0121] Further, based on the above embodiment, please refer to Fig.11 In one of the exemplary embodiments provided in the present application, the specific implementation process of the above-mentioned driving warning method may also include step S1110 and step S1120, which are described in detail as follows:
[0122] Step S1110, performing perspective projection transformation on the camera coordinate system to obtain an image coordinate system, so as to obtain a pixel coordinate system corresponding to the real-time video image based on the image coordinate system;
[0123] Step S1120, generating a normalized imaging plane corresponding to the real-time image based on the property parameters corresponding to the mobile terminal and the pixel coordinate system.
[0124] The camera coordinate system is a three-dimensional rectangular coordinate system whose origin is usually located at the optical center of the camera (also called the focusing center or focus), the Z axis coincides with the optical axis of the camera and points to the front of the camera (that is, perpendicular to the imaging plane), and the X axis and Y axis are parallel to the imaging plane.
[0125] Through the internal and external parameters obtained during the camera calibration process, we can determine the camera coordinate system of the camera. Specifically, the external parameters of the camera (rotation matrix and translation vector) describe the transformation relationship between the camera coordinate system and the world coordinate system. Using these external parameters, we can transform points in the world coordinate system into the camera coordinate system.
[0126] Specifically, the extrinsic parameters usually describe the position and posture of the camera (i.e., the camera in the mobile terminal) in the world coordinate system. These parameters include the rotation matrix and the translation vector, which define the transformation from the world coordinate system to the camera coordinate system. The camera coordinate system is determined by the physical structure and installation position of the camera, and the extrinsic parameters describe the relationship between the camera coordinate system and the world coordinate system. In fact, the extrinsic parameters are used to convert points in the world coordinate system to the camera coordinate system in subsequent calculations.
[0127] It should be noted that perspective projection transformation is a basic concept in computer vision, which describes how points in three-dimensional space are projected onto a two-dimensional image plane. In this process, points in the camera coordinate system (usually three-dimensional) are converted to the image coordinate system (usually two-dimensional). This transformation involves the intrinsic parameters of the camera, such as focal length, optical center, etc., as well as the perspective projection model of the camera (such as the pinhole camera model). Through perspective projection transformation, the image coordinate system corresponding to the real-time video image can be obtained.
[0128] Among them, to generate the normalized imaging plane, the intrinsic parameters of the camera and the pixel coordinate system are needed. The pixel coordinate system is the representation of the image in the computer memory, which defines the position of each point in the image in pixels. The normalized imaging plane (also called the imaging plane in the normalized image coordinate or camera coordinate system) is a virtual plane that is located in front of the camera and has a specific distance (usually the focal length) from the camera coordinate system. On this plane, all points from the world coordinate system are projected into two-dimensional, normalized coordinates. The x- and y-components of these coordinates are in the range of [-1,1], and the scale differences caused by different intrinsic parameters of the camera (such as focal length) are removed.
[0129] Specifically, we can first transform the points in the pixel coordinate system into the image physical coordinate system (i.e., the coordinate system after taking into account the pixel size), and then project these points onto the normalized imaging plane through the camera's focal length and other parameters. In this process, the property parameters (which may refer to parameters related to the physical characteristics of the camera, such as pixel size, focal length, etc.) play a key role.
[0130] like Fig.12 As shown, Fig.12 This is a projection process from the original image pixels to the world coordinate system shown in an exemplary embodiment of the present application. The internal and external parameters jointly establish the mapping relationship between the real world and the original image, such as Fig.12 , from right to left is the imaging process from the real world to the original image, and from left to right is the projection process from the original image pixels to the real world coordinate system.
[0131] In this embodiment, the perspective projection transformation can accurately simulate the process of the camera lens projecting the three-dimensional world onto the two-dimensional image plane, thereby ensuring the accuracy of the image coordinate system, and the precise pixel coordinates and normalized imaging plane provide a basis for high-precision positioning and measurement tasks, such as three-dimensional reconstruction and target tracking.
[0132] Further, based on the above embodiment, in one of the exemplary embodiments provided in the present application, the specific implementation process of the above driving warning method may also include the following steps, which are described in detail as follows:
[0133] The external parameters of the mobile terminal are determined based on the internal parameters of the mobile terminal, the coordinate position of the mobile terminal, the vanishing point in the original image, and the angle of the horizontal line.
[0134] Specifically, in order to enable the recognition of objects in the early warning system to be converted from the image coordinate system (pixels) to the vehicle coordinate system (inside) to obtain usable environmental perception data, the calibration system needs to further calibrate the external parameters of the camera of the mobile terminal. The calibration process can further optimize the external parameters through the internal parameters of the mobile terminal, the coordinate position of the mobile terminal in the vehicle coordinate system, and the horizontal angle between the horizontal line in the mobile terminal and the standard horizon.
[0135] Further, based on the above embodiments, please refer to Fig.13 In one of the exemplary embodiments provided in the present application, the specific implementation method of the above-mentioned driving warning method may also include steps S1310 to S1330, which are described in detail as follows:
[0136] Step S1310, determining a relative rotation and translation matrix of the mobile terminal relative to the calibration coordinate system based on the intrinsic parameters of the mobile terminal, the vanishing points in the original image, and the angle of the horizontal line;
[0137] Step S1320, obtaining a rotation and translation matrix of the mobile terminal in the vehicle coordinate system based on the coordinate position of the mobile terminal;
[0138] Step S1330, determining the extrinsic parameters of the mobile terminal based on the relative rotation and translation matrix and the rotation and translation matrix in the vehicle coordinate system.
[0139] Specifically, the external parameter calibration can be based on the internal parameter K, the position of the mobile terminal relative to the vehicle coordinate system (x car ,y car , z car ), vanishing point coordinates (X v , Y v ) and the horizontal angle θ h For example, the attitude matrix R of the terminal camera relative to the calibration coordinate system can be obtained according to the following formula: cali2cam , since the backward camera has no displacement relative to the calibration coordinate system, the rotation and translation matrix is R cali2cam .
[0140]
[0141] R cali2cam =R z *R y *R x
[0142]
[0143] RT car2cali =RT car2cali *RT cali2cam
[0144]
[0145] In this embodiment, by utilizing image features such as vanishing points and horizontal line angles, combined with internal parameters, the rotation and translation matrix of the mobile terminal relative to the calibration coordinate system can be accurately calculated, which can effectively reduce the positioning error caused by factors such as image distortion and noise. By combining the relative rotation and translation matrix with the coordinate position of the mobile terminal in the vehicle coordinate system, the precise position and direction of the mobile terminal in the vehicle coordinate system can be further determined, providing a reliable basis for subsequent image processing and analysis.
[0146] Fig.14 It is a brief flow diagram of refreshing driving warning in an exemplary application scenario. In the application scenario shown in 14, the mobile terminal is installed in the adjustable area of the vehicle, and the coordinate position of the vehicle installation position in the vehicle coordinate corresponding to the vehicle is determined, and then the real-time video image corresponding to the real world collected by the mobile terminal is obtained; the corresponding horizontal line in the real-time video image and the standard horizon corresponding to the real-time video image are obtained; the horizontal angle between the horizontal line and the standard horizon is calculated, so as to determine the horizontal posture deflection of the mobile terminal in the calibration coordinate system based on the horizontal angle. The vanishing point on the real-time video image is obtained, and the vanishing point is the intersection point between the parallel lines in the real-time video image; the coordinate offset of the vanishing point in the calibration coordinate system is calculated, and the lateral posture deflection and longitudinal posture offset of the mobile terminal in the calibration coordinate system are determined based on the coordinate offset. Based on the posture deflection of the mobile terminal in the calibration coordinate system, the effective area corresponding to the vanishing point and the horizontal line in the real-time video image is generated; based on the effective area, the installation position of the mobile terminal is adjusted so that the posture of the mobile terminal is consistent with the vehicle coordinate system. Obtain parameter information corresponding to the real-time video image, the parameter information includes the height and width of the image; determine the center point and focal length corresponding to the real-time video image based on the width and height of the image; determine the internal and external parameters corresponding to the mobile terminal based on the center point and focal length, wherein the internal and external parameters are used to establish a mapping relationship between the world coordinate system and the vehicle coordinate system; determine the distance parameter between the real world content in the effective image based on the mapping relationship, and generate a warning strategy for the vehicle based on the distance parameter. For the detailed implementation process, please refer to the records in the aforementioned embodiments, which will not be repeated here.
[0147] Fig.15 is a block diagram of a vehicle warning device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown in the figure is specifically configured in the server 120. The device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applied.
[0148] like Fig.15 As shown, the exemplary driving warning device includes: an acquisition module 1510, used to acquire the installation position corresponding to the mobile terminal, the mobile terminal being installed in an adjustable area of the vehicle; a determination module 1520, used to determine the coordinate position corresponding to the mobile terminal in the vehicle coordinate system based on the vehicle coordinate system corresponding to the vehicle, the vehicle coordinate system being used to characterize the relative position relationship between the real world and the vehicle; an adjustment module 1530, used to acquire the real-time video image corresponding to the real world captured by the mobile terminal, and adjust the installation position of the mobile terminal based on the real-time video image and the coordinate position so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system; a construction module 1540, used to construct a warning strategy for the vehicle based on the effective image captured by the adjusted mobile terminal and the performance parameters corresponding to the mobile terminal, so as to control the vehicle to execute the warning strategy when the effective image meets the warning trigger condition in the warning strategy.
[0149] According to one aspect of an embodiment of the present application, the above-mentioned adjustment module 1530 is also used to obtain real-time video images collected by the mobile terminal within a continuous time period; if the real-time video image is not within the valid area, the posture deflection of the coordinate position in the calibration coordinate system is calculated, and the origin of the calibration coordinate system is the coordinate position corresponding to the mobile terminal in the vehicle coordinate system; the installation position of the mobile terminal is adjusted based on the posture deflection.
[0150] According to one aspect of an embodiment of the present application, the above-mentioned adjustment module 1530 is also used to obtain the corresponding horizontal line in the real-time video image and the standard horizon corresponding to the real-time video image; calculate the horizontal angle between the horizontal line and the standard horizon, so as to determine the horizontal posture deflection amount of the mobile terminal in the calibrated coordinate system based on the horizontal angle.
[0151] According to one aspect of an embodiment of the present application, the above-mentioned adjustment module 1530 is also used to obtain a vanishing point on a real-time video image, where the vanishing point is the intersection point between parallel lines in the real-time video image; calculate the coordinate offset of the vanishing point in a calibrated coordinate system, and determine the lateral posture deflection and longitudinal posture offset of the mobile terminal in the calibrated coordinate system based on the coordinate offset.
[0152] According to one aspect of an embodiment of the present application, the above-mentioned adjustment module 1530 is also used to generate effective areas corresponding to vanishing points and horizontal lines in real-time video images based on the posture deflection amount of the mobile terminal in the calibrated coordinate system; and adjust the installation position of the mobile terminal based on the effective area so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system.
[0153] According to one aspect of an embodiment of the present application, the above-mentioned construction module 1540 is also used to obtain parameter information corresponding to the real-time video image, the parameter information including the height and width of the image; determine the center point and focal length corresponding to the real-time video image based on the width and height of the image; determine the internal and external parameters corresponding to the mobile terminal based on the center point and focal length, wherein the internal and external parameters are used to establish a mapping relationship between the world coordinate system and the vehicle coordinate system; determine the distance parameters between the real-world content in the effective image based on the mapping relationship, and generate a warning strategy for the vehicle based on the distance parameters.
[0154] According to one aspect of an embodiment of the present application, the above-mentioned construction module 1540 is also used to obtain a distortion coefficient corresponding to the mobile terminal, and perform distortion processing on the real-time video image based on the distortion coefficient to obtain an undistorted image; convert the undistorted image into an undistorted original image based on the intrinsic parameters corresponding to the mobile terminal, the intrinsic parameters include performance parameters of the mobile terminal, wherein the intrinsic parameters are used to establish a mapping relationship between the camera coordinate system and the pixel coordinate system of the mobile terminal; convert the undistorted original image into an image in the vehicle coordinate system based on the extrinsic parameters corresponding to the mobile terminal, wherein the extrinsic parameters are used to establish a mapping relationship between the camera coordinate system and the world coordinate system; establish a mapping relationship between the vehicle coordinate system and the world coordinate system based on the coordinate position, the mapping relationship between the camera coordinate system and the pixel coordinate system, and the mapping relationship between the camera coordinate system and the world coordinate system.
[0155] According to one aspect of an embodiment of the present application, the above-mentioned construction module 1540 is also used to perform perspective projection transformation on the camera coordinate system to obtain an image coordinate system, so as to obtain a pixel coordinate system corresponding to the real-time video image based on the image coordinate system; and to produce a normalized imaging plane corresponding to the real-time image based on the property parameters corresponding to the mobile terminal and the pixel coordinate system.
[0156] According to one aspect of an embodiment of the present application, the construction module 1540 is further used to determine the external parameters of the mobile terminal based on the internal parameters of the mobile terminal, the coordinate position of the mobile terminal, the vanishing point in the original image, and the angle of the horizontal line.
[0157] According to one aspect of an embodiment of the present application, the above-mentioned construction module 1540 is also used to determine the relative rotation and translation matrix of the mobile terminal relative to the calibration coordinate system based on the intrinsic parameters of the mobile terminal, the vanishing point in the original image and the angle of the horizontal line; obtain the rotation and translation matrix of the mobile terminal in the vehicle coordinate system based on the coordinate position of the mobile terminal; and determine the external parameters of the mobile terminal based on the relative rotation and translation matrix and the rotation and translation matrix in the vehicle coordinate system.
[0158] It should be noted that the traffic warning device provided in the above embodiment and the traffic warning method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the traffic warning device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0159] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the traffic warning method provided in the above-mentioned embodiments.
[0160] Fig.16 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.16 The computer system 1600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0161] like Fig.16 As shown, the computer system 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1602 or the program loaded from the storage part 1608 to the random access memory (RAM) 1603, such as executing the method described in the above embodiment. In RAM 1603, various programs and data required for system operation are also stored. CPU 1601, ROM 1602 and RAM 1603 are connected to each other through bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.
[0162] The following components are connected to the I / O interface 1605: an input section 1606 including a keyboard, a mouse, etc.; an output section 1607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as needed. A removable medium 1611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1610 as needed, so that a computer program read therefrom is installed into the storage section 1608 as needed.
[0163] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1609, and / or installed from a removable medium 1611. When the computer program is executed by a central processing unit (CPU) 1601, various functions defined in the system of the present application are executed.
[0164] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0165] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0166] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0167] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the driving warning method as described above is implemented. The computer-readable storage medium can be included in the electronic device described in the above embodiment, or it can exist independently without being assembled into the electronic device.
[0168] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the driving warning method provided in each of the above embodiments.
[0169] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.
Claims
1. A driving warning method, characterized in that: include: Acquire an installation position corresponding to a mobile terminal, wherein the mobile terminal is installed in an adjustable area of a vehicle; Determining the coordinate position of the mobile terminal in the vehicle coordinate system based on the vehicle coordinate system corresponding to the vehicle, wherein the vehicle coordinate system is used to represent the relative position relationship between the real world and the vehicle; Acquire a real-time video image corresponding to the real world captured by the mobile terminal, and adjust the installation position of the mobile terminal based on the real-time video image and the coordinate position so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system; The early warning strategy of the vehicle is constructed based on the effective image collected by the adjusted mobile terminal and the performance parameters corresponding to the mobile terminal, so as to control the vehicle to execute the early warning strategy when the effective image meets the early warning trigger condition in the early warning strategy.
2. The method according to claim 1, characterized in that The acquiring a real-time video image corresponding to the real world captured by the mobile terminal, and adjusting the installation position of the mobile terminal based on the real-time video image and the coordinate position, comprises: Acquire real-time video images collected by the mobile terminal within a continuous time period; If the real-time video image is not within the effective area, then calculating the attitude deflection amount of the coordinate position in the calibration coordinate system, the calibration coordinate system origin of the calibration coordinate system being the coordinate position corresponding to the mobile terminal in the vehicle coordinate system; The installation position of the mobile terminal is adjusted based on the posture deflection amount.
3. The method according to claim 2, characterized in that The method further comprises: Acquire a corresponding horizontal line in the real-time video image and a standard horizon corresponding to the real-time video image; Calculating a horizontal angle between the horizontal line and the standard horizon, so as to determine a horizontal posture deflection amount of the mobile terminal in the calibration coordinate system based on the horizontal angle; Acquire a vanishing point on the real-time video image, where the vanishing point is an intersection point between parallel lines in the real-time video image; The coordinate offset of the vanishing point in the calibration coordinate system is calculated, and the lateral posture deflection and the longitudinal posture offset of the mobile terminal in the calibration coordinate system are determined based on the coordinate offset.
4. The method according to claim 3, characterized in that The method further comprises: Generate the effective areas corresponding to the vanishing points and the horizontal lines in the real-time video image based on the posture deflection amount of the mobile terminal in the calibration coordinate system; The installation position of the mobile terminal is adjusted based on the effective area so that the posture of the mobile terminal is consistent with that of the vehicle coordinate system.
5. The method according to claim 4, characterized in that The constructing of the vehicle warning strategy based on the adjusted valid image collected by the mobile terminal and the performance parameters corresponding to the mobile terminal includes: Obtain parameter information corresponding to the real-time video image, the parameter information including the height and width of the image; Determine the center point and focal length corresponding to the real-time video image based on the width and height of the image; Determining internal and external parameters corresponding to the mobile terminal based on the center point and the focal length, wherein the internal and external parameters are used to establish a mapping relationship between a world coordinate system and a vehicle coordinate system; A distance parameter between real-world contents in the effective image is determined based on the mapping relationship, and a warning strategy for the vehicle is generated based on the distance parameter.
6. The method according to claim 5, characterized in that The method comprises: Acquiring a distortion coefficient corresponding to the mobile terminal, and performing distortion processing on the real-time video image based on the distortion coefficient to obtain an undistorted image; Converting the undistorted image into an undistorted original image based on intrinsic parameters corresponding to the mobile terminal, the intrinsic parameters comprising performance parameters of the mobile terminal, wherein the intrinsic parameters are used to establish a mapping relationship between a camera coordinate system of the mobile terminal and the pixel coordinate system; Converting the undistorted original image into an image in the vehicle coordinate system based on extrinsic parameters corresponding to the mobile terminal, wherein the extrinsic parameters are used to establish a mapping relationship between the camera coordinate system and the world coordinate system; A mapping relationship between the vehicle coordinate system and the world coordinate system is established based on the coordinate position, a mapping relationship between the camera coordinate system and the pixel coordinate system, and a mapping relationship between the camera coordinate system and the world coordinate system.
7. The method according to claim 6, characterized in that The method further comprises: Performing perspective projection transformation on the camera coordinate system to obtain an image coordinate system, so as to obtain a pixel coordinate system corresponding to the real-time video image based on the image coordinate system; A normalized imaging plane corresponding to the real-time image is generated based on the property parameters corresponding to the mobile terminal and the pixel coordinate system.
8. The method according to claim 6, characterized in that The method further comprises: Determine a relative rotation and translation matrix of the mobile terminal relative to the calibration coordinate system based on the intrinsic parameters of the mobile terminal, the vanishing point in the original image, and the horizontal line angle; Obtaining a rotation and translation matrix of the mobile terminal in the vehicle coordinate system based on the coordinate position of the mobile terminal; The extrinsic parameters of the mobile terminal are determined based on the relative rotation and translation matrix and the rotation and translation matrix in the vehicle coordinate system.
9. A driving warning device, characterized in that: The device comprises: An acquisition module, used to acquire an installation position corresponding to a mobile terminal, wherein the mobile terminal is installed in an adjustable area of a vehicle; A determination module, configured to determine a coordinate position of the mobile terminal in the vehicle coordinate system based on a vehicle coordinate system corresponding to the vehicle, wherein the vehicle coordinate system is used to represent a relative position relationship between the real world and the vehicle; an adjustment module, configured to obtain a real-time video image corresponding to the real world captured by the mobile terminal, and adjust the installation position of the mobile terminal based on the real-time video image and the coordinate position so that the posture of the mobile terminal is consistent with the vehicle coordinate system; A construction module is used to construct a warning strategy for the vehicle based on the effective image collected by the adjusted mobile terminal and the performance parameters corresponding to the mobile terminal, so as to control the vehicle to execute the warning strategy when the effective image meets the warning trigger condition in the warning strategy.
10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the driving warning method as described in any one of claims 1 to 8.
