A driving assistance image processing method, device, equipment and storage medium

Abstract

The application discloses a driving assistance image processing method, device and equipment and a storage medium, and comprises the following steps: in response to the vehicle being in a starting stage, a standard coordinate system with the vehicle as the origin is established; image data of cameras arranged on both sides of the vehicle is continuously acquired, the acquired image data is matched with the standard coordinate system, and a vehicle frame and road obstacles are identified in the image data; according to the two-dimensional position coordinates of the identified vehicle frame and road obstacles in the corresponding image data, the three-dimensional position coordinates of the vehicle frame and road obstacles in the standard coordinate system are converted and generated; according to the three-dimensional position coordinates of the vehicle frame and road obstacles in the standard coordinate system, the distance between the vehicle frame and road obstacles is calculated; and the image data labeled with the vehicle frame, road obstacles and the distance therebetween is sequentially fed back to a vehicle display, so that the processing of the driving assistance image is completed.

Description

Technical Field

[0001] This invention relates to the field of automotive intelligent technology, and in particular to a driving assistance image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of automobiles, the number of cars on the road is increasing, leading to a growing demand for car functions. In addition, due to the different driving skills of drivers, auxiliary functions such as cruise control, reversing camera, panoramic camera, and side camera have emerged to help drivers drive better, reduce driver workload, or reduce car ownership costs.

[0003] Currently, the development of automotive assistance functions, from rearview mirrors to panoramic imaging, provides observation of the surrounding environment from multiple angles. However, it does not support the calculation and intuitive display of the distance to obstacles on both sides. It only passively monitors with ultrasonic radar. At this point, the driver is already very close to the obstacle, and the space for the driver to adjust the vehicle is limited. Moreover, existing panoramic and side images are stitched images, which have problems such as distortion, blurred edges, excessively close angles, and errors in image timing. They cannot realistically, intuitively, and in real time show obstacles on both sides in front of the vehicle, and whether there is a risk of scraping the side of the vehicle.

[0004] Therefore, there is an urgent need for a method that can improve the accuracy of driving assistance image display and can realistically, intuitively and in real time show obstacles on both sides in front. Summary of the Invention

[0005] This invention provides a driving assistance image processing method, device, equipment, and storage medium to solve the technical problems in the prior art, such as distortion, blurred edges, excessively close angles, and errors in image timing, which prevent the realistic, intuitive, and real-time display of obstacles on both sides in front.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a driving assistance image processing method, comprising:

[0007] In response to the vehicle being in the startup phase, a standard coordinate system is established with the vehicle as the origin; the standard coordinate system is a three-dimensional coordinate system.

[0008] The system continuously acquires image data from cameras located on both sides of the vehicle, pairs the acquired image data with the standard coordinate system, and identifies the vehicle outline and road obstacles in the image data.

[0009] Based on the two-dimensional position coordinates of the identified vehicle outline and road obstacles in the corresponding image data, the three-dimensional position coordinates of the vehicle outline and road obstacles in the standard coordinate system are transformed and generated.

[0010] The distance between the vehicle frame and the road obstacle is calculated based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system.

[0011] The image data, which are marked with the vehicle outline, road obstacles and the distances between them, are fed back to the vehicle's display in turn, thereby completing the processing of driving assistance images.

[0012] As a preferred embodiment, in response to the vehicle being in the startup phase, a standard coordinate system is established with the vehicle as the origin, specifically as follows:

[0013] In response to the vehicle being started, the cameras located on both sides of the vehicle are initialized.

[0014] The camera is used to acquire test images after initialization, and the vehicle outline in the test images is identified.

[0015] Based on the percentage of the vehicle frame's area in the test image, the acquisition angles of the cameras on both sides of the vehicle are controlled so that the vehicle frame occupies a preset percentage of the test image in each camera.

[0016] After controlling the cameras, determine the final acquisition angle of each camera, and based on the final acquisition angle of each camera, determine the intersection point of the cameras in the horizontal direction of the vehicle, which will serve as the origin of the vehicle, and establish a standard coordinate system.

[0017] As a preferred embodiment, the continuous acquisition of image data from cameras positioned on both sides of the vehicle, and the pairing of the acquired image data with the standard coordinate system, specifically involves:

[0018] According to a preset acquisition time interval, image data from cameras located on both sides of the vehicle after control is terminated is continuously acquired. In the first acquisition phase, the first set of image data acquired initially and its adjacent second set of image data are used as the first paired image. The first paired image and the standard coordinate system are calibrated based on the final acquisition angle of each camera, the vehicle's origin, and the viewing angle of the first paired image. The first paired image includes the first set of image data acquired initially through the cameras on both sides and the second set of image data adjacent to the first set of image data.

[0019] In the second acquisition phase, the currently acquired image data is compared with the previous set of image data for difference detection. Based on the difference information obtained from the difference detection, the currently acquired image data is paired into the standard coordinate system, thereby enabling the acquisition of the next set of image data. Continuous acquisition includes the first acquisition phase and the second acquisition phase.

[0020] As a preferred embodiment, identifying the vehicle outline and road obstacles in the image data specifically involves:

[0021] After each set of image data is collected, the system uses a preset recognition model to collect features from the collected image data, thereby identifying vehicle borders and road obstacles based on the collected feature information.

[0022] As a preferred embodiment, the step of converting and generating the three-dimensional position coordinates of the vehicle outline and road obstacles in the standard coordinate system based on the identified two-dimensional position coordinates of the vehicle outline and road obstacles in the corresponding image data specifically involves:

[0023] Based on the collected feature information, the two-dimensional position coordinates of the vehicle frame and road obstacles are obtained;

[0024] Based on the three-dimensional coordinate system paired with the current image data, the two-dimensional position coordinates are calibrated and transformed according to the two-dimensional position coordinates in the current image data, thereby obtaining the three-dimensional position coordinates of the vehicle frame and road obstacles in the three-dimensional coordinate system.

[0025] As a preferred embodiment, the step of calculating the distance between the vehicle frame and the road obstacle based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system specifically involves:

[0026] The distance between the vehicle frame and the road obstacle is calculated based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system, as well as the preset scale for image data acquisition by the camera.

[0027] As a preferred option, it also includes:

[0028] Continuously acquire image data from a camera installed on the roof of the vehicle, as roof image data;

[0029] Based on the preset position of the vehicle's roof camera, the vehicle's roof image data and the standard coordinate system are calibrated in the standard coordinate system.

[0030] In the roof image data, the roof frame and the top obstacle are identified respectively, and the first position coordinates of the roof frame and the top obstacle in the roof image data are determined respectively. Then, the second position coordinates of the roof frame and the top obstacle in the standard coordinate system are transformed and generated respectively.

[0031] The distance between the roof frame and the top obstacle is calculated based on the second position coordinates of the roof frame and the top obstacle in the standard coordinate system.

[0032] The roof image data, which is marked with the roof edge, top obstacles and the distances between them, is fed back to the vehicle's display.

[0033] Accordingly, the present invention also provides a driving assistance image processing device, comprising: a coordinate system module, an acquisition module, a coordinate positioning module, a calculation module, and a display module;

[0034] The coordinate system module is used to establish a standard coordinate system with the vehicle as the origin in response to the vehicle being in the start-up phase; the standard coordinate system is a three-dimensional coordinate system.

[0035] The acquisition module is used to continuously acquire image data from cameras set on both sides of the vehicle, pair the acquired image data with the standard coordinate system, and identify the vehicle frame and road obstacles in the image data.

[0036] The coordinate positioning module is used to convert and generate the three-dimensional position coordinates of the vehicle frame and road obstacles in the standard coordinate system based on the two-dimensional position coordinates of the identified vehicle frame and road obstacles in the corresponding image data.

[0037] The calculation module is used to calculate the distance between the vehicle frame and the road obstacle based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system.

[0038] The display module is used to sequentially feed back image data marked with vehicle outlines, road obstacles and the distances between them to the vehicle display, thereby completing the processing of driving assistance images.

[0039] Accordingly, the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the driving assistance image processing method as described above.

[0040] Accordingly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the driving assistance image processing method as described above.

[0041] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0042] The technical solution of this invention establishes a standard coordinate system when the vehicle is in the startup phase, enabling the standard coordinate system to be established in real time according to the current state of the vehicle. This improves the accuracy of subsequent image data pairing and calibration. Then, by acquiring image data from cameras set on both sides of the vehicle, the vehicle frame and road obstacles are identified. Furthermore, by transforming the two-dimensional position coordinates of the vehicle frame and road obstacles in the image data into three-dimensional coordinates in the standard coordinate system, the distance between the vehicle frame and the road obstacles is calculated. The labeled image data of the vehicle frame, road obstacles, and the distance between them are then fed back to the vehicle's infotainment system for auxiliary display. This avoids the need to stitch together the collected image information and use radar to achieve obstacle recognition. Instead, it provides auxiliary judgment by displaying the real images and distances on both sides of the vehicle, achieving an intuitive, direct, and real-time display of obstacles in front on both sides. Attached Figure Description

[0043] Figure 1 : A flowchart illustrating the steps of a driving assistance image processing method provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of driving assistance images on both sides of a vehicle provided in an embodiment of the present invention;

[0045] Figure 3 : A schematic diagram of a vehicle roof-mounted driving assistance image provided in an embodiment of the present invention.

[0046] Figure 4 : This is a schematic diagram of the structure of a driving assistance image processing device provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Example 1

[0049] Please refer to Figure 1 The present invention provides a driving assistance image processing method, comprising the following steps S101-S105:

[0050] Step S101: In response to the vehicle being in the startup phase, establish a standard coordinate system with the vehicle as the origin; the standard coordinate system is a three-dimensional coordinate system.

[0051] As a preferred embodiment, the step of establishing a standard coordinate system with the vehicle as the origin in response to the vehicle being in the starting phase specifically involves:

[0052] In response to the vehicle being started, the cameras positioned on both sides of the vehicle are initialized; test images are acquired from the initialized cameras, and the vehicle border in the test images is identified; based on the percentage of the display area occupied by the vehicle border in the test image, the acquisition angle of the cameras on both sides of the vehicle is controlled so that the vehicle border occupies a preset percentage in the test image in each camera; after the camera control is terminated, the final acquisition angle of each camera is determined, and based on the final acquisition angle of each camera, the intersection point of the cameras in the horizontal direction of the vehicle is determined as the origin of the vehicle, and a standard coordinate system is established.

[0053] In this embodiment, during the vehicle startup phase, the cameras on both sides of the vehicle need to be initialized to ensure that the camera positions can be calibrated each time the vehicle starts. This allows for adjustments to the cameras to capture accurate external images in real time, thereby improving the accuracy of driving assistance image data processing.

[0054] It should be noted that, preferably, the initialization process of the camera is as follows: the camera angle is controlled so that the camera is 10° outward and 20° downward in the direction parallel to the vehicle body, thereby ensuring that the camera is in a position that can recognize the vehicle sound frame and road obstacles.

[0055] In this embodiment, the initialized camera also needs to acquire test images to avoid significant differences in image data acquired by the camera at a fixed angle due to differences in vehicle height or body length. Furthermore, by acquiring the test image of the camera after initialization, the vehicle body in the test image is identified, and its border is marked. This allows determination of whether the test image at the current viewpoint falls within a reasonable field of view. Preferably, the reasonableness of the current viewpoint is judged by the percentage of the display area occupied by the vehicle border in the entire test image. For example, the preset percentage can be set to 20%-30% based on the actual vehicle model. When the vehicle border occupies 20%-30% of the test image, it indicates that the vehicle border will not affect the camera's acquisition of road obstacles, thus allowing for a direct and intuitive display of obstacles on both sides in front.

[0056] In this embodiment, the horizontal and vertical movement directions of the camera corresponding to the test image are calculated based on the percentage of the display area of the vehicle frame in the entire test image and the position information of the vehicle frame in the test image. Exemplarily, by evenly dividing the test image into a "field" shape, for the camera installed on the left side of the vehicle, if the percentage of the display area of the vehicle frame in the entire test image is too large, it means that the vehicle frame exceeds the area divided by the lower right corner of the test image. That is, it is necessary to control the camera to move left in the horizontal direction and up in the vertical direction. After calculating the horizontal and vertical movement directions of the camera, the image data is collected once every time the camera moves 1° in the horizontal direction or 1° in the vertical direction, and then a judgment is made again until the vehicle frame is in the area divided by the lower right corner (corresponding to the left camera) or the lower left corner (corresponding to the right camera) of the test image.

[0057] Further, for the left camera, after the vehicle frame is in the area divided by the lower right corner of the test image, the current angle of the left camera is recorded and stored as the final acquisition angle, so that when the vehicle is in the startup stage next time, the left camera can be initialized and controlled according to the final acquisition angle. Similarly, for the right camera, the same operation process is adopted.

[0058] In this embodiment, according to the final acquisition angles corresponding to the cameras on both sides of the vehicle, the intersection point of the cameras in the horizontal direction of the vehicle is determined. Since the cameras are generally symmetrically installed on both sides of the vehicle, the intersection point of the extension lines of the cameras in the horizontal direction can be used as the origin of the entire vehicle, and then a three-dimensional standard coordinate system is established. Preferably, the forward direction of the vehicle coincides with the bisector of the x-axis and the y-axis, the left side of the vehicle body is the x-axis, the right side of the vehicle body is the y-axis, and the vehicle height corresponds to the z-axis.

[0059] Step S102: Continuously obtain the image data of the cameras installed on both sides of the vehicle, pair the obtained image data with the standard coordinate system, and identify the vehicle frame and road obstacles in the image data.

[0060] As a preferred solution of this embodiment, the continuous obtaining of the image data of the cameras installed on both sides of the vehicle and pairing the obtained image data with the standard coordinate system is specifically as follows:

[0061] According to a preset acquisition time interval, image data from cameras located on both sides of the vehicle after control is terminated is continuously acquired. In the first acquisition phase, the first set of image data acquired initially and its adjacent second set are used as the first paired image. The first paired image and the standard coordinate system are calibrated based on the final acquisition angle of each camera, the vehicle's origin, and the viewing angle of the first paired image. The first paired image includes the first set of image data acquired initially through the cameras on both sides and the second set of image data adjacent to the first set. In the second acquisition phase, the currently acquired image data is compared with the previous set of image data using difference detection. Based on the difference information obtained from the difference detection, the currently acquired image data is paired into the standard coordinate system, thereby enabling the acquisition of the next set of image data. Continuous acquisition includes both the first and second acquisition phases.

[0062] In this embodiment, by dividing the entire image data acquisition process into a first acquisition stage and a second acquisition stage, it is possible to ensure that different image data undergo different calibration processes with the standard coordinate axis at different acquisition stages. In the first acquisition stage, it is possible to ensure that all first paired images are accurately calibrated with the standard coordinate system. However, since the calibration calculation requires a large amount of computing power and resources, the second acquisition stage is entered after calibrating the first set of image data acquired for the first time and its adjacent second set of image data. At the same time, since the difference between the image data acquired at the previous moment and the image data acquired at the next moment is smaller when the preset acquisition time interval is smaller for image data acquired by the same camera, the difference between the currently acquired image data and the previous set of image data can be detected, and the currently acquired image data can be paired with the standard coordinate system based on the difference information obtained from the difference detection, thereby reducing the amount of computation and improving the efficiency of image processing.

[0063] Furthermore, to avoid inaccurate calibration information due to prolonged pairing of image data during the second acquisition phase, the vehicle's status can be detected. For example, if the vehicle is in Park (P) or has been parked for longer than a preset time (e.g., waiting at a red light, in traffic jam, temporary parking), the system can re-enter the first acquisition phase to recalibrate the image data collected by the camera. After calibration, the system can re-enter the second acquisition phase to pair the image data collected by the camera. This alternation between the first and second acquisition phases improves the accuracy between the camera image data and the standard coordinate system.

[0064] In this embodiment, the calibration of the first paired image with the standard coordinate system can be achieved through camera calibration technology. By determining the standard coordinate system, the camera's position in the standard coordinate system (the position of the camera on the vehicle), and the camera's angle, the acquired first paired image can be calibrated with the standard coordinate system. Furthermore, pairing the currently acquired image data with the standard coordinate system can be achieved through a 3D image reconstruction and fusion difference information fusion algorithm and image feature point calibration. Simultaneously, since the preset acquisition time interval can be set sufficiently small, the differences between the image data acquired before and after are minimal. Therefore, feature analysis and matching can be directly performed on the image data acquired before and after, locating feature points in the image data. Then, multiple boundary points are located along the periphery of the feature using these feature points. Multiple boundary points are fitted along the periphery of the feature using the fitted boundary lines, and a perspective transformation matrix is ​​generated based on the feature points. This achieves the goal of calibrating the previous image data with the standard coordinate system and then pairing the subsequent image data with the standard coordinate system.

[0065] In a preferred embodiment, the step of identifying the vehicle outline and road obstacles in the image data specifically involves:

[0066] After each set of image data is collected, the system uses a preset recognition model to collect features from the collected image data, thereby identifying vehicle borders and road obstacles based on the collected feature information.

[0067] In this embodiment, the preset recognition model is trained by pre-training vehicle bounding box images and road obstacle images. Preferably, the preset recognition model includes, but is not limited to, models such as AlexNet, VGG19, ResNet_152, InceptionV4, and DenseNet.

[0068] Step S103: Based on the two-dimensional position coordinates of the identified vehicle frame and road obstacles in the corresponding image data, convert and generate the three-dimensional position coordinates of the vehicle frame and road obstacles in the standard coordinate system.

[0069] As a preferred embodiment, the step of converting and generating the three-dimensional position coordinates of the vehicle outline and road obstacles in the standard coordinate system based on the identified two-dimensional position coordinates of the vehicle outline and road obstacles in the corresponding image data specifically involves:

[0070] Based on the collected feature information, the two-dimensional position coordinates of the vehicle frame and road obstacles are obtained; based on the three-dimensional coordinate system paired with the current image data, the two-dimensional position coordinates are calibrated and transformed according to the two-dimensional position coordinates in the current image data, thereby obtaining the three-dimensional position coordinates of the vehicle frame and road obstacles in the three-dimensional coordinate system.

[0071] In this embodiment, since the image data is paired or calibrated on a standard coordinate system in step S102, the two-dimensional position coordinates of the vehicle frame and road obstacles on the corresponding image data can be obtained and converted to the standard coordinate system, thereby realizing the conversion between two-dimensional position coordinates and three-dimensional coordinate system.

[0072] Step S104: Calculate the distance between the vehicle frame and the road obstacle based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system.

[0073] In a preferred embodiment, the step of calculating the distance between the vehicle frame and the road obstacle based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system specifically involves:

[0074] The distance between the vehicle frame and the road obstacle is calculated based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system, as well as the preset scale for image data acquisition by the camera.

[0075] In this embodiment, the actual distance between the vehicle frame and the road obstacle can be calculated by using the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system, combined with the preset scale set by the camera when the image data is acquired.

[0076] Step S105: The image data marked with the vehicle outline, road obstacles and the distance between them are fed back to the vehicle display in sequence, thereby completing the processing of driving assistance images.

[0077] In this embodiment, the distances on both sides are calibrated to mark the constant display scale of the side front images, or to support real-time calculation of the distances to obstacles on both sides and display them in the left and right side front image modes of the large screen AVM. The lateral distance between the obstacle and the vehicle is displayed in real time to help the driver determine the distance of the obstacle from the sides of the vehicle. When the distance of the object from the side of the vehicle (including the predicted distance) is <100cm, the system is triggered and can be manually turned off. After being turned off, the current trigger will no longer be displayed.

[0078] Furthermore, as another preferred option, please refer to Figure 2The vehicle's outline is marked with a horizontal, fixed line. Preferably, the direction of the fixed line is determined by the most prominent points of the outer front and rear wheel arches, but this can be customized as needed. Road obstacles are marked as lines on the side closest to the obstacle, displayed horizontally as moving lines that can vary according to the shape of the road or obstacle. For example, the line can be displayed in real-time based on the distance between the object and the vehicle's side, and can be curved to a certain extent depending on the shape of the object. The moving line is determined by two points parallel to the vehicle and the point closest to the vehicle. When the distance is <50cm, a 1Hz alarm sounds; <30cm, a 2Hz alarm sounds and a reminder to pay attention to the rearview mirror position; and <10cm, a 3Hz alarm sounds.

[0079] It is understandable that this embodiment adopts a direct display method, without stitching, that is, it does not combine the images with those from the rear-view or side-view cameras. Compared with the stitching method, the direct display method has the advantage of no image distortion, which is closer to the actual observation effect of the driver. The image data transmission principle is as follows: the camera detects the environmental image, transmits it to the transmission line, then to the domain controller, and finally to the large screen. When the driving assistance image is manually turned on, or when the driving assistance image is automatically triggered in a narrow scene, the large screen displays the driving environment in front on the vehicle's large screen in full-screen or full-screen mode, as well as the corresponding soft-switch display mode.

[0080] As a preferred embodiment, it also includes:

[0081] The system continuously acquires image data from a camera mounted on the vehicle roof, using this data as roof image data. Based on the preset position of the camera on the vehicle roof, the system calibrates the roof image data and the standard coordinate system. Within the roof image data, the system identifies the roof frame and top obstacles, determining their first position coordinates within the data, and then converting and generating their second position coordinates in the standard coordinate system. Based on these second position coordinates, the system calculates the distance between the roof frame and the top obstacle. Finally, the roof image data, labeled with the roof frame, top obstacle, and the distance between them, is sequentially fed back to the vehicle's infotainment system.

[0082] In this embodiment, a camera installed on the roof of the vehicle can determine its coordinate position in a standard coordinate system. Then, by acquiring image data, it can identify the top edge and the top obstacle, thereby determining the first position coordinates. Then, it can convert and generate the second position coordinates of the roof edge and the top obstacle in the standard coordinate system, and finally calculate the distance between the roof edge and the top obstacle, so as to realize the roof collision display and warning.

[0083] Preferably, please refer to Figure 3 The lower guideline is positioned on the roof edge and is fixed in place. Further, when calibrating the lower guideline, a preset distance directly in front (which can be determined based on the location of the roof camera and the preset scale of the camera), preferably 4m, is used as the calibrator / line at the same height as the roof to calibrate and fix the display position of the lower guideline in this imaging mode. The upper guideline is positioned on the top obstacle and is movable. A calibrator / line 6m directly in front and 2m high is used to calibrate the highest position of the upper guideline, and it is gradually moved down 3cm to calibrate other heights within the range. The alarm trigger logic depends on the vehicle's height: when the distance is <50cm, a 1Hz alarm sounds; <30cm, a 2Hz alarm sounds and a reminder to pay attention to roof safety; and <10cm, a 3Hz alarm sounds.

[0084] Implementing the above embodiments has the following effects:

[0085] The technical solution of this invention establishes a standard coordinate system when the vehicle is in the startup phase, enabling the standard coordinate system to be established in real time according to the current state of the vehicle. This improves the accuracy of subsequent image data pairing and calibration. Then, by acquiring image data from cameras set on both sides of the vehicle, the vehicle frame and road obstacles are identified. Furthermore, by transforming the two-dimensional position coordinates of the vehicle frame and road obstacles in the image data into three-dimensional coordinates in the standard coordinate system, the distance between the vehicle frame and the road obstacles is calculated. The labeled image data of the vehicle frame, road obstacles, and the distance between them are then fed back to the vehicle's infotainment system for auxiliary display. This avoids the need to stitch together the collected image information and use radar to achieve obstacle recognition. Instead, it provides auxiliary judgment by displaying the real images and distances on both sides of the vehicle, achieving an intuitive, direct, and real-time display of obstacles in front on both sides.

[0086] Example 2

[0087] Please see Figure 4 The present invention provides a driving assistance image processing device, comprising: a coordinate system module 201, an acquisition module 202, a coordinate positioning module 203, a calculation module 204, and a display module 205.

[0088] The coordinate system module 201 is used to establish a standard coordinate system with the vehicle as the origin in response to the vehicle being in the start-up phase; the standard coordinate system is a three-dimensional coordinate system.

[0089] The acquisition module 202 is used to continuously acquire image data from cameras set on both sides of the vehicle, pair the acquired image data with the standard coordinate system, and identify the vehicle frame and road obstacles in the image data.

[0090] The coordinate positioning module 203 is used to convert and generate the three-dimensional position coordinates of the vehicle frame and road obstacles in the standard coordinate system based on the two-dimensional position coordinates of the identified vehicle frame and road obstacles in the corresponding image data.

[0091] The calculation module 204 is used to calculate the distance between the vehicle frame and the road obstacle based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system.

[0092] The display module 205 is used to sequentially feed back image data marked with vehicle outlines, road obstacles and the distances between them to the vehicle display, thereby completing the processing of driving assistance images.

[0093] As a preferred embodiment, in response to the vehicle being in the startup phase, a standard coordinate system is established with the vehicle as the origin, specifically as follows:

[0094] In response to the vehicle being started, the cameras positioned on both sides of the vehicle are initialized; test images are acquired from the initialized cameras, and the vehicle border in the test images is identified; based on the percentage of the display area occupied by the vehicle border in the test image, the acquisition angle of the cameras on both sides of the vehicle is controlled so that the vehicle border occupies a preset percentage in the test image in each camera; after the camera control is terminated, the final acquisition angle of each camera is determined, and based on the final acquisition angle of each camera, the intersection point of the cameras in the horizontal direction of the vehicle is determined as the origin of the vehicle, and a standard coordinate system is established.

[0095] As a preferred embodiment, the continuous acquisition of image data from cameras positioned on both sides of the vehicle, and the pairing of the acquired image data with the standard coordinate system, specifically involves:

[0096] According to a preset acquisition time interval, image data from cameras located on both sides of the vehicle after control is terminated is continuously acquired. In the first acquisition phase, the first set of image data acquired initially and its adjacent second set are used as the first paired image. The first paired image and the standard coordinate system are calibrated based on the final acquisition angle of each camera, the vehicle's origin, and the viewing angle of the first paired image. The first paired image includes the first set of image data acquired initially through the cameras on both sides and the second set of image data adjacent to the first set. In the second acquisition phase, the currently acquired image data is compared with the previous set of image data using difference detection. Based on the difference information obtained from the difference detection, the currently acquired image data is paired into the standard coordinate system, thereby enabling the acquisition of the next set of image data. Continuous acquisition includes both the first and second acquisition phases.

[0097] As a preferred embodiment, identifying the vehicle outline and road obstacles in the image data specifically involves:

[0098] After each set of image data is collected, the system uses a preset recognition model to collect features from the collected image data, thereby identifying vehicle borders and road obstacles based on the collected feature information.

[0099] As a preferred embodiment, the step of converting and generating the three-dimensional position coordinates of the vehicle outline and road obstacles in the standard coordinate system based on the identified two-dimensional position coordinates of the vehicle outline and road obstacles in the corresponding image data specifically involves:

[0100] Based on the collected feature information, the two-dimensional position coordinates of the vehicle frame and road obstacles are obtained; based on the three-dimensional coordinate system paired with the current image data, the two-dimensional position coordinates are calibrated and transformed according to the two-dimensional position coordinates in the current image data, thereby obtaining the three-dimensional position coordinates of the vehicle frame and road obstacles in the three-dimensional coordinate system.

[0101] As a preferred embodiment, the step of calculating the distance between the vehicle frame and the road obstacle based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system specifically involves:

[0102] The distance between the vehicle frame and the road obstacle is calculated based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system, as well as the preset scale for image data acquisition by the camera.

[0103] As a preferred option, a roof-mounted recognition module 206 is also included:

[0104] The roof recognition module 206 is used to continuously acquire image data from a camera installed on the roof of the vehicle as roof image data; based on the preset position of the roof camera on the vehicle, the roof image data and the standard coordinate system are calibrated in the standard coordinate system; in the roof image data, the roof frame and the top obstacle are identified respectively, and the first position coordinates of the roof frame and the top obstacle in the roof image data are determined, and then converted and generated as second position coordinates of the roof frame and the top obstacle in the standard coordinate system; based on the second position coordinates of the roof frame and the top obstacle in the standard coordinate system, the distance between the roof frame and the top obstacle is calculated; and the roof image data marked with the roof frame, the top obstacle and the distance between them are sequentially fed back to the vehicle display.

[0105] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] Implementing the above embodiments has the following effects:

[0107] The technical solution of this invention establishes a standard coordinate system when the vehicle is in the startup phase, enabling the standard coordinate system to be established in real time according to the current state of the vehicle. This improves the accuracy of subsequent image data pairing and calibration. Then, by acquiring image data from cameras set on both sides of the vehicle, the vehicle frame and road obstacles are identified. Furthermore, by transforming the two-dimensional position coordinates of the vehicle frame and road obstacles in the image data into three-dimensional coordinates in the standard coordinate system, the distance between the vehicle frame and the road obstacles is calculated. The labeled image data of the vehicle frame, road obstacles, and the distance between them are then fed back to the vehicle's infotainment system for auxiliary display. This avoids the need to stitch together the collected image information and use radar to achieve obstacle recognition. Instead, it provides auxiliary judgment by displaying the real images and distances on both sides of the vehicle, achieving an intuitive, direct, and real-time display of obstacles in front on both sides.

[0108] Example 3

[0109] Accordingly, the present invention also provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the driving assistance image processing method as described in any of the above embodiments.

[0110] The terminal device in this embodiment includes a processor, a memory, and a computer program and computer instructions stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps described in Embodiment 1 above, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiment, such as the coordinate positioning module 203.

[0111] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device. For example, the coordinate positioning module 203 is used to convert and generate the three-dimensional position coordinates of the identified vehicle outline and road obstacles in the standard coordinate system based on the two-dimensional position coordinates of the identified vehicle outline and road obstacles in the corresponding image data.

[0112] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0113] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0114] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0115] Wherein, if the modules / units integrated in the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0116] Example 4

[0117] Accordingly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the driving assistance image processing method as described in any of the above embodiments.

[0118] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A driving assistance image processing method, characterized in that, include: In response to the vehicle being in the startup phase, a standard coordinate system is established with the vehicle as the origin. The standard coordinate system is a three-dimensional coordinate system; The system continuously acquires image data from cameras positioned on both sides of the vehicle and pairs the acquired image data with the standard coordinate system. Specifically, this includes: continuously acquiring image data from cameras positioned on both sides of the vehicle after control has ended, according to a preset acquisition time interval; wherein, in the first acquisition phase, the first set of image data acquired initially and its adjacent second set of image data are used as the first paired image, and the first paired image and the standard coordinate system are calibrated according to the final acquisition angle of each camera, the origin of the vehicle, and the viewing angle position of the first paired image; the first paired image includes the first set of image data acquired initially through the cameras on both sides and the second set of image data adjacent to the first set of image data; in the second acquisition phase, the currently acquired image data is compared with the previous set of image data for difference detection, and the currently acquired image data is paired with the standard coordinate system based on the difference information obtained from the difference detection, thereby enabling the acquisition of the next set of image data; continuous acquisition includes the first acquisition phase and the second acquisition phase; and vehicle outlines and road obstacles are identified in the image data. Based on the two-dimensional position coordinates of the identified vehicle outline and road obstacles in the corresponding image data, the three-dimensional position coordinates of the vehicle outline and road obstacles in the standard coordinate system are transformed and generated. The distance between the vehicle frame and the road obstacle is calculated based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system. The image data, which are marked with the vehicle outline, road obstacles and the distances between them, are fed back to the vehicle's display in turn, thereby completing the processing of driving assistance images.

2. The driving assistance image processing method as described in claim 1, characterized in that, In response to the vehicle being in the startup phase, a standard coordinate system is established with the vehicle as the origin, specifically as follows: In response to the vehicle being started, the cameras located on both sides of the vehicle are initialized. The camera is used to acquire test images after initialization, and the vehicle outline in the test images is identified. Based on the percentage of the vehicle frame's area in the test image, the acquisition angles of the cameras on both sides of the vehicle are controlled so that the vehicle frame occupies a preset percentage of the test image in each camera. After controlling the cameras, determine the final acquisition angle of each camera, and based on the final acquisition angle of each camera, determine the intersection point of the cameras in the horizontal direction of the vehicle, which will serve as the origin of the vehicle, and establish a standard coordinate system.

3. The driving assistance image processing method as described in claim 2, characterized in that, The process of identifying vehicle outlines and road obstacles in the image data specifically involves: After each set of image data is collected, the system uses a preset recognition model to collect features from the collected image data, thereby identifying vehicle borders and road obstacles based on the collected feature information.

4. The driving assistance image processing method as described in claim 3, characterized in that, The step of converting and generating the three-dimensional position coordinates of the vehicle outline and road obstacles in the standard coordinate system based on the identified two-dimensional position coordinates of the vehicle outline and road obstacles in the corresponding image data is as follows: Based on the collected feature information, the two-dimensional position coordinates of the vehicle frame and road obstacles are obtained; Based on the three-dimensional coordinate system paired with the current image data, the two-dimensional position coordinates are calibrated and transformed according to the two-dimensional position coordinates in the current image data, thereby obtaining the three-dimensional position coordinates of the vehicle frame and road obstacles in the three-dimensional coordinate system.

5. The driving assistance image processing method as described in claim 4, characterized in that, The distance between the vehicle frame and the road obstacle is calculated based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system, specifically as follows: The distance between the vehicle frame and the road obstacle is calculated based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system, as well as the preset scale for image data acquisition by the camera.

6. The driving assistance image processing method as described in claim 5, characterized in that, Also includes: Continuously acquire image data from a camera installed on the roof of the vehicle, as roof image data; Based on the preset position of the vehicle's roof camera, the vehicle's roof image data and the standard coordinate system are calibrated in the standard coordinate system. In the roof image data, the roof frame and the top obstacle are identified respectively, and the first position coordinates of the roof frame and the top obstacle in the roof image data are determined respectively. Then, the second position coordinates of the roof frame and the top obstacle in the standard coordinate system are transformed and generated respectively. The distance between the roof frame and the top obstacle is calculated based on the second position coordinates of the roof frame and the top obstacle in the standard coordinate system. The roof image data, which is marked with the roof edge, top obstacles and the distances between them, is fed back to the vehicle's display.

7. A driving assistance image processing device, characterized in that, include: The module includes a coordinate system module, an acquisition module, a coordinate positioning module, a calculation module, and a display module. The coordinate system module is used to establish a standard coordinate system with the vehicle as the origin in response to the vehicle being in the startup phase; the standard coordinate system is a three-dimensional coordinate system. The acquisition module is used to continuously acquire image data from cameras located on both sides of the vehicle and pair the acquired image data with the standard coordinate system. Specifically, it includes: continuously acquiring image data from cameras located on both sides of the vehicle after control has ended, according to a preset acquisition time interval; wherein, in the first acquisition phase, the first set of image data acquired initially and its adjacent second set of image data are used as the first paired image, and the first paired image and the standard coordinate system are calibrated according to the final acquisition angle of each camera, the origin of the vehicle, and the viewing angle position of the first paired image; the first paired image includes the first set of image data acquired initially through the cameras on both sides and the second set of image data adjacent to the first set of image data; in the second acquisition phase, the currently acquired image data is compared with the previous set of image data for difference detection, and the currently acquired image data is paired with the standard coordinate system based on the difference information obtained from the difference detection, thereby enabling the acquisition of the next set of image data; continuous acquisition includes the first acquisition phase and the second acquisition phase; and vehicle outlines and road obstacles are identified in the image data. The coordinate positioning module is used to convert and generate the three-dimensional position coordinates of the vehicle frame and road obstacles in the standard coordinate system based on the two-dimensional position coordinates of the identified vehicle frame and road obstacles in the corresponding image data. The calculation module is used to calculate the distance between the vehicle frame and the road obstacle based on the three-dimensional position coordinates of the vehicle frame and the road obstacle in the standard coordinate system. The display module is used to sequentially feed back image data marked with vehicle outlines, road obstacles and their distances to the vehicle display, thereby completing the processing of driving assistance images.

8. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the driving assistance image processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the driving assistance image processing method as described in any one of claims 1 to 6.

Images