Road Image Display Method, Device, and Storage Medium

Abstract

An embodiment of the present application discloses a method, device, and storage medium for road image display, belonging to the technical field of automotive assisted driving. In the embodiment of the present application, the first road image sent by the controller in the ADAS includes the marking information of multiple targets detected by the controller. On this basis, the information of the targets included in the second road image obtained based on the marking information of the multiple targets and the first road image will be more complete, reducing the probability of missing display or misdisplay of the road image, and thus being able to better assist the driver in driving the vehicle.

Description

Technical Field

[0001] This application relates to the technical field of automotive assisted driving, and particularly relates to a road image display method, device, and storage medium. Background Art

[0002] With the development of automotive assisted driving technology, the installation rate of ADAS (Advanced Driver Assistance System) in vehicles is getting higher and higher.

[0003] In related technologies, the controller in ADAS can obtain the road image collected by the front camera of the vehicle, extract the road condition parameters from the road image, and then transmit the extracted road condition parameters to the display node. After receiving the road condition parameters, the display node generates a simulated road image for display according to the road condition parameters.

[0004] In the above processing method, the road condition parameters extracted by the controller in ADAS may be inaccurate, which will cause the road information in the simulated road image generated by the display node to not match the road information in the road image collected by the camera, resulting in display errors. Summary of the Invention

[0005] Embodiments of this application provide a road image display method, device, and storage medium, which can reduce the probability of missing display or misdisplay of information in the road image, thereby better assisting the driver in driving the vehicle. The technical solution is as follows:

[0006] On the one hand, a road image display method is provided. The method includes:

[0007] Receiving a first road image sent by a controller in ADAS, where the first road image includes the marking information of multiple targets detected by the controller;

[0008] Based on the marking information of the multiple targets and the first road image, obtaining a second road image;

[0009] Displaying the second road image.

[0010] Optionally, the obtaining a second road image based on the marking information of the multiple targets and the first road image includes:

[0011] Based on the marking information of the multiple targets, superimposing the image models of the multiple targets on the first road image to obtain a third road image;

[0012] Based on the third road image, obtaining the second road image.

[0013] Optionally, superimposing image models of the multiple targets in the first road image based on the marking information of the multiple targets includes:

[0014] Cropping the first road image based on the calibrated size;

[0015] Acquire label information of each target in the cropped first road image;

[0016] Based on the acquired label information of each target, an image model of the corresponding target is superimposed on the cropped first road image.

[0017] Optionally, the marking information of each target includes a target type, an image position, and an angle of the corresponding target;

[0018] The step of superimposing an image model of the corresponding target on the cropped first road image based on the acquired label information of each target includes:

[0019] acquiring, based on an object type of a first object, an image model corresponding to the corresponding object, wherein the first object is any object in the cropped first road image;

[0020] An image model of the first target is overlaid at the image position of the first target in the cropped first road image, and an angle of the image model of the first target is adjusted based on the angle of the first target.

[0021] Optionally, acquiring the second road image based on the third road image includes:

[0022] removing background information from the third road image;

[0023] The alarm information is added to the third road image after the background information is removed to obtain the second road image.

[0024] In another aspect, a road image display device is provided, the device comprising:

[0025] a receiving module, configured to receive a first road image sent by a controller in the ADAS, wherein the first road image includes marking information of a plurality of targets detected by the controller;

[0026] an acquisition module, configured to acquire a second road image based on the marking information of the plurality of targets and the first road image;

[0027] A display module is configured to display the second road image.

[0028] Optionally, the acquisition module is used to:

[0029] Based on the marking information of the multiple targets, superimpose the image models of the multiple targets on the first road image to obtain a third road image;

[0030] Based on the third road image, obtain the second road image.

[0031] Optionally, the obtaining module is further configured to:

[0032] Crop the first road image based on the calibrated size;

[0033] Obtain the marking information of each target in the cropped first road image;

[0034] Based on the obtained marking information of each target, superimpose the corresponding target image model on the cropped first road image.

[0035] Optionally, the marking information of each target includes the target type, image position, and angle of the corresponding target;

[0036] The obtaining module is further configured to:

[0037] Based on the target type of the first target, obtain the corresponding target image model, where the first target is any target in the cropped first road image;

[0038] Cover the image model of the first target at the image position of the first target in the cropped first road image, and adjust the angle of the image model of the first target based on the angle of the first target.

[0039] Optionally, the obtaining module is further configured to:

[0040] Remove the background information from the third road image;

[0041] Add alarm information to the third road image after removing the background information to obtain the second road image.

[0042] On the other hand, a computer-readable storage medium is provided, in which a computer program is stored, and when the computer program is executed by a computer, the steps of the above-mentioned road image display method are implemented.

[0043] On the other hand, a computer program product including instructions is provided, and when it runs on a computer, it causes the computer to execute the steps of the above-mentioned road image display method.

[0044] The beneficial effects brought by the technical solution provided by the embodiments of the present application at least include:

[0045] In the embodiment of the present application, the first road image sent by the controller in the ADAS includes the marking information of multiple targets detected by the controller. On this basis, the information of the targets included in the second road image obtained based on the marking information of the multiple targets and the first road image will be more complete, reducing the probability of missing display or misdisplay of the road image, so as to better assist the driver in driving the vehicle. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0047] Figure 1 It is a system architecture diagram related to a road image display method provided by an embodiment of the present application;

[0048] Figure 2 It is a flowchart of a road image display method provided by an embodiment of the present application;

[0049] Figure 3 It is a schematic diagram of the effective area in a first road image provided by an embodiment of the present application;

[0050] Figure 4 It is a schematic diagram of a second road image provided by an embodiment of the present application;

[0051] Figure 5 It is a schematic diagram of a road image display device provided by an embodiment of the present application;

[0052] Figure 6 It is a schematic diagram of the structure of a display device provided by an embodiment of the present application. Detailed Embodiments

[0053] To make the purpose, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the drawings.

[0054] Before explaining the embodiments of the present application in detail, the system architecture related to the embodiments of the present application will be introduced.

[0055] Figure 1 It is a system architecture diagram related to a road image display method provided by an embodiment of the present application. As Figure 1As shown in the figure, the system includes an in-vehicle camera 101, a controller 102 in the ADAS, and a display node 103. Among them, communication between the in-vehicle camera 101 and the controller 102 in the ADAS can be carried out through an LVDS (Low-Voltage Differential Signaling) bus or a coaxial cable, and communication between the controller 102 in the ADAS and the display node 103 can be carried out through an LVDS bus, a coaxial cable, an Ethernet, or a USB (Universal Serial Bus).

[0056] Among them, the in-vehicle camera 101 can collect road images within its own field of view and transmit the collected road images to the controller 102 in the ADAS through an LVDS bus or a coaxial cable.

[0057] The controller 102 in the ADAS can receive the road images collected by the in-vehicle camera 101. After receiving the road images, the controller 102 in the ADAS can detect each target in the road images and mark the detected targets, thereby obtaining a first road image. The first road image includes the marking information of multiple targets detected by the controller. Subsequently, the controller 102 in the ADAS can transmit the first road image to the display node 103 through an LVDS bus, a coaxial cable, an Ethernet, or a USB.

[0058] The display node 103 is used to receive the first road image sent by the controller 102 in the ADAS, and based on the first road image and the marking information of multiple targets included in the first road image, obtain a second road image and display the second road image.

[0059] Among them, the in-vehicle camera 101 can be a front camera of the vehicle, and the front camera can be deployed at the front end of the vehicle body. The embodiments of the present application do not make any limitations in this regard.

[0060] The controller 102 in the ADAS can be a domain controller on the vehicle, and the display node 103 can be an instrument or a central control host on the vehicle. The embodiments of the present application do not make any limitations in this regard.

[0061] Next, the road image display method provided by the embodiments of the present application will be introduced.

[0062] Figure 2 This is a road image display method provided by the embodiments of the present application. This method can be applied to the display nodes introduced above, such as Figure 2 As shown in the figure, this method includes the following steps:

[0063] Step 201: Receive a first road image sent by a controller in the ADAS. The first road image includes marking information of multiple targets detected by the controller.

[0064] In an embodiment of the present application, the controller in the ADAS can obtain a road image collected by an in-vehicle camera, and detect each target in the road image. Then, mark each detected target to obtain a first road image. Among them, the first road image includes marking information of multiple detected targets. Then, the controller in the ADAS can send the first road image to a display node. The display node receives the first road image. Among them, the multiple targets include vehicles, pedestrians, obstacles, lane lines, traffic lights, etc. in the road.

[0065] Exemplarily, after the controller in the ADAS receives the road image collected by the in-vehicle camera, it can input the road image into a road target detection model to detect each target included in the road image. Then, add marking information of each target to the road image according to the detection result output by the road target detection model.

[0066] Among them, the road target detection model is a model pre-trained with a large number of road images. Based on this, when the road target detection model receives a road image, it can process the received road image to detect each target included in the road image, and then output the detection result of each target in the road image. Among them, the detection result includes multiple target detection frames, each target detection frame includes a target, and each target detection frame corresponds to the target type of the included target. Among them, the target types of each target in the road image can be vehicles, pedestrians, obstacles, lane lines, traffic lights, etc.

[0067] Then, the controller in the ADAS can mark the target types of each target in the road image based on the detection result output by the road target detection model.

[0068] Exemplarily, the controller in the ADAS can mark "vehicle" on the vehicle in the road image and "pedestrian" on the "pedestrian" to indicate the target types of the corresponding targets. Since the lane lines included in the road image may be double yellow lines, single yellow lines, white solid lines, white dotted lines, etc., the controller in the ADAS can mark the lane lines with the corresponding colors and line shapes according to the color and line shape of the lane lines included in the road image.

[0069] After marking the target types of each target in the road image, the controller in the ADAS can mark the image positions and angles of each target in the road image.

[0070] Exemplarily, the controller in the ADAS can establish a first image coordinate system in the road image. The origin of the first image coordinate system can be the lower left corner point of the road image. The X-axis is parallel to the width direction of the road image, and the positive direction is to the right. The Y-axis is parallel to the height direction of the road image, and the positive direction is upward. For the convenience of explanation, the image position coordinates of any point in the first image coordinate system are denoted as (x, y). Based on this, the controller in the ADAS can determine the image position coordinates of the center points of each target in the first image coordinate system based on the corner points of each target detection box in the detection result output by the road target detection model. For example, taking any target detection box in the detection result as an example, the controller in the ADAS can connect the corner points at the diagonal positions in the target detection box to obtain two diagonals, and then use the intersection point of the two diagonals as the center point of the target. The controller in the ADAS can determine the image position coordinates of the center point of the target in the first image coordinate system, and then mark the image position coordinates of the center point of the target, and use the image position coordinates of the center point of the target in the first image coordinate system as the image position of the target. According to the same method, the controller in the ADAS can mark the image position coordinates of the center points of multiple targets in the road image.

[0071] It should be noted that for each lane line included in the road image, the controller in the ADAS can determine the starting point, ending point, and intermediate key points of each lane line in the road image, and mark the image position coordinates of the starting point, ending point, and intermediate key points of each lane line in the first image coordinate system, and use the image position coordinates of the starting point, ending point, and intermediate key points of each lane line in the first image coordinate system as the image position of each lane line. Among them, the intermediate key point can be the turning point of the lane line.

[0072] Exemplarily, the controller in the ADAS can also determine the angle of the corresponding target in the road image according to the corner points of each target detection box in the detection result output by the road target detection model. Similarly, taking any target detection box in the detection result as an example, the controller in the ADAS can connect two adjacent corner points along the length direction of the target detection box, and then determine the included angle between the connection line of the two corner points and the X-axis, and then use this included angle as the angle of the target in the target detection box in the road image. Then mark the angle of the target in the road image. According to the same method, the controller in the ADAS can mark the angle of each target in the road image.

[0073] Optionally, after marking the object type, image position, and angle of each object in the first road image, the controller in the ADAS may further generate an identification code for each object and carry the identification code in the marking information of the corresponding object, wherein the identification code is used to uniquely identify an object.

[0074] The controller in the ADAS can then use the marked road image as a first road image and send the first road image to a display node, which can receive the first road image. The first road image includes marking information for multiple objects, where the marking information for each object includes the object type, image position, angle, and identification code of the corresponding object.

[0075] It should be noted that the resolution of the first road image sent by the controller in the ADAS to the display node is fixed. Based on this, the controller in the ADAS can compress and encode the first road image based on the resolution of the first road image. The compressed first road image is then transmitted to the display node via an LVDS bus, coaxial cable, Ethernet, or USB. After receiving the compressed first road image, the display node can decode the first road image to obtain the first road image.

[0076] Step 202: Acquire a second road image based on the marking information of the plurality of objects and the first road image.

[0077] In this embodiment of the present application, after receiving the first road image, the display node may overlay image models of the multiple objects on the first road image based on their labeling information to obtain a third road image. The display node may then obtain the second road image based on the third road image.

[0078] Exemplarily, the display node may crop the first road image based on the calibrated size; obtain marking information of each target in the cropped first road image; and then, based on the obtained marking information of each target, superimpose an image model of the corresponding target in the cropped first road image.

[0079] It should be noted that a calibration size is pre-stored in the display node, and the calibration size is used to calibrate the effective area in the received first road image.

[0080] Exemplarily, after receiving the first road image, the display node can establish a first image coordinate system in the first road image. The origin of the coordinate system of the first image is the lower left corner point of the first road image. The X-axis is parallel to the width direction of the first road image, and the positive direction is to the right. The Y-axis is parallel to the height direction of the first road image, and the positive direction is upward. Among them, it is assumed that the resolution of the first road image received by the display node is a×b, and the pre-stored calibration size in the display node is m×n. In this case, the display node can calibrate the effective area in the first road image according to the pre-stored calibration size and the resolution of the first road image.

[0081] Among them, the display node calculates the difference between the horizontal size in the calibration size and the horizontal size in the resolution of the first road image. Calculate half of the difference to obtain a first value. Starting from the left boundary of the first road image, determine the effective left boundary whose distance from the left boundary is the first value. Starting from the right boundary of the first road image, determine the effective right boundary whose distance from the right boundary is the first value. Using the same method, the effective upper boundary and the effective lower boundary can also be determined. The area enclosed by the effective left boundary, the effective right boundary, the effective upper boundary and the effective lower boundary is used as the effective area in the first road image.

[0082] For example, as Figure 3 shown, it is assumed that the resolution of the first road image is 800×450, and the pre-stored calibration size is 600×300. Based on this, the display node can calibrate the area where X≥100, X≤700, Y≥75, and Y≤375 in the first road image based on the calibration size.

[0083] Of course, the display node can also determine the effective area based on other principles. For example, the effective upper boundary and the effective lower boundary can be moved up or down, and the effective left boundary and the effective right boundary can be moved left or right, etc. The embodiments of the present application do not limit this.

[0084] After that, the display node can crop the part other than the effective area based on the effective area calibrated in the first road image to obtain the cropped first road image. Among them, the cropped first road image is actually the effective area. After that, the display node can obtain the marking information of each target in the cropped first road image.

[0085] It should be noted that the first road image includes the marking information of multiple targets. Some targets may be located outside the effective area determined above, and some targets are located within the effective area. Based on this, after obtaining the cropped first road image, the display node can obtain the marking information of the targets located within the effective area, that is, obtain the marking information of each target in the cropped first road image.

[0086] Among them, the display node can use the size of the cropped first road image and the resolution of the first road image to determine the coordinate transformation relationship between the image coordinate system where the cropped first road image is located and the image coordinate system where the first road image is located. Furthermore, based on the image position of each target in the first image coordinate system and this coordinate transformation relationship, the image position of each target in the cropped first road image is determined.

[0087] Exemplarily, the display node can establish a second image coordinate system in the cropped first road image. Among them, the origin of coordinates of the second image coordinate system is the lower left corner point of the cropped first road image, the X-axis is parallel to the width direction of the cropped first road image, and the positive direction is to the right, and the Y-axis is parallel to the height direction of the cropped first road image, and the positive direction is upward. For the convenience of distinction, the image position coordinates of any point in the second image coordinate system are denoted as (X, Y). From the foregoing introduction, the resolution of the first road image can be a×b, and the resolution of the cropped first road image can be m×n. Based on this, the coordinate transformation relationship between the second image coordinate system where the cropped first road image is located and the first image coordinate system where the first road image is located can be represented by the following formulas (1) and (2).

[0088]

[0089]

[0090] Based on this, taking any target in the cropped first road image as an example, assuming that the image position coordinates of the target in the first image coordinate system are (x1, y1), the display node can determine the image position coordinates (X1, Y1) of the target in the second image coordinate system in the cropped first road image based on the following formulas (3) and (4).

[0091]

[0092]

[0093] After that, the display node can use the image position coordinates of the target in the second image coordinate system in the cropped first road image as the image position of the target in the cropped first road image. Then, use this image position and the target type, angle, and identification code in the marking information of the target in the cropped first road image as the marking information of the target in the cropped first road image. According to the same method, the display node can obtain the marking information of each target in the cropped first road image.

[0094] Among them, based on the above formulas (3) and (4), the display node can also determine the image position coordinates of the key points marked on each lane line included in the cropped first road image in the second image coordinate system.

[0095] After obtaining the marking information of each target in the cropped first road image, the display node can superimpose the image model of the corresponding target on the cropped first road image based on the obtained marking information of each target.

[0096] Exemplarily, taking any target included in the cropped first road image as an example, for the sake of convenience of explanation, this target is called the first target. The display node can obtain the image model corresponding to the first target based on the target type of the first target. Then, cover the image model of the first target at the image position of the first target in the cropped first road image, and adjust the angle of the image model of the first target based on the angle of the first target.

[0097] It should be noted that the display node pre-stores the mapping relationship between the image models of multiple targets and the target types. The image models of the multiple targets can be 3D models of the multiple targets. Among them, the image models of the multiple targets can include the image models of vehicles, pedestrians, traffic lights, and obstacles, etc.

[0098] Among them, the display node can, based on the target type of the first target, search in the mapping relationship between the image models and the target types stored in itself for the target type that is the same as the target type of the first target, and then use the image model corresponding to the found target type as the image model of the first target.

[0099] After obtaining the image model corresponding to the first target, the display node can, based on the image position of the first target in the cropped first road image, that is, the image position coordinates of the first target in the second image coordinate system, cover the image model of the first target at the image position of the first target in the cropped first road image. Then, adjust the angle of the image model of the first target based on the angle of the first target so that the angle of the image model of the first target is consistent with the angle of the first target in the cropped first road image. According to the same method, the display node can superimpose the image models of the corresponding targets on the cropped first road image according to the marking information of each target in the cropped first road image.

[0100] It should be noted that for road markings such as lane lines included in the first road image after cropping, if the road marking has a corresponding image model, the above method can be used to overlay the corresponding image model. If the road marking does not have a corresponding image model, for example, the line direction of the lane line may vary and cannot be represented by a single image model. In such a case, the display node can draw the road marking at the corresponding image position based on the image position of the road marking in the cropped first road image. For example, for lane lines, the display node can draw the lane lines according to the image positions of the starting point, ending point, and key points of the lane lines in the cropped first road image.

[0101] Optionally, in another possible implementation, after receiving the first road image, the display node may not crop the first road image but directly obtain the marking information of each target in the first road image, and then overlay the image model of the corresponding target in the first road image based on the obtained marking information of each target. The implementation process of overlaying the image model of the corresponding target in the first road image based on the marking information of each target in the first road image can refer to the method introduced above, and this application embodiment will not elaborate on it anymore.

[0102] It should be noted that in the embodiments of this application, the size of the resolution of the images that the display node can display is preset in the display node. In this case, the display node can adjust the resolution of the cropped first road image overlaid with the image model, or the first road image overlaid with the image model according to the size of the resolution of the images that it can display, so as to obtain the third road image.

[0103] Among them, the display node can generate a blank image according to the size of the resolution of the images that it can display, and then convert the road image overlaid with the image model into the blank image to obtain the third road image.

[0104] Exemplarily, assume that the resolution of the images that the display node can display is p×q, then the resolution of the blank image is also p×q. Among them, the pixel value corresponding to each pixel point in the blank image is a specified pixel value, for example, it can be 0 or 255.

[0105] After generating the blank image, the display node can establish a third image coordinate system in the blank image. The origin of the coordinate system of the third image is the lower left corner point of the blank image. The X-axis is parallel to the width direction of the blank image, and the positive direction is to the right. The Y-axis is parallel to the height direction of the blank image, and the positive direction is upward. For the convenience of distinction, the image position coordinates of any point in the third image coordinate system are denoted as (x', y'). Among them, if the above-mentioned is the image model covering the corresponding target in the cropped first road image, the display node can first determine the coordinate conversion relationship between the third image coordinate system where the blank image is located and the second image coordinate system where the cropped first road image is located. As can be seen from the above introduction, the resolution of the cropped first road image can be m×n, and the resolution of the blank image generated by the display node is p×q. Based on this, the coordinate conversion relationship between the third image coordinate system and the second image coordinate system can be expressed by the following formulas (5) and (6).

[0106]

[0107]

[0108] After that, the display node can convert the image position coordinates of each pixel point in the cropped first road image according to the coordinate conversion relationship between the second image coordinate system and the third image coordinate system shown in the above formulas (5) and (6), so as to obtain the image position coordinates of the corresponding target position points of each pixel point in the blank image. On this basis, the display node can use the pixel value of each pixel point in the cropped first road image as the pixel value of the corresponding target position point of the pixel point in the blank image, so as to realize the conversion of the cropped first road image to the blank image to obtain the third road image.

[0109] Optionally, in another possible implementation, if the above-mentioned is the image model covering the corresponding target in the first road image, in this case, the display node can convert the first road image covering the image model of the corresponding target to the blank image to obtain the third road image.

[0110] Exemplarily, the display node can first determine the coordinate conversion relationship between the third image coordinate system where the blank image is located and the first image coordinate system where the first road image is located. As can be seen from the above introduction, the resolution of the first road image can be a×b, and the resolution of the blank image can be p×q. Based on this, the coordinate conversion relationship between the third image coordinate system and the first image coordinate system can be expressed by the following formulas (7) and (8).

[0111]

[0112]

[0113] After determining the coordinate transformation relationship between the third image coordinate system and the first image coordinate system, the display node can transform the image position coordinates of each pixel in the first road image based on this coordinate transformation relationship, thereby obtaining the image position coordinates of the target location corresponding to each pixel in the blank image. Based on this, the display node can use the pixel value of each pixel in the first road image as the pixel value of the target location corresponding to that pixel in the blank image, thereby transforming the first road image into the blank image to obtain the third road image.

[0114] After acquiring the third road image, the display node may acquire the second road image based on the third road image.

[0115] As can be seen from the above description, the third road image is obtained by overlaying image models of multiple objects within the cropped or uncropped first road image. That is, the third road image includes multiple layers, including the cropped or uncropped first road image. In this embodiment of the present application, the cropped or uncropped first road image included in the third road image can be used as background information. Based on this, the display node can remove the background information from the third road image and use the third road image after the background information is removed as the second road image.

[0116] Alternatively, in some other possible implementations, the display node may also receive warning information for each target sent by the ADAS controller, where the warning information carries the target's identification code. Based on this, after removing background information from the third road image, the display node may also add the warning information to the background-removed third road image to obtain the second road image.

[0117] It should be noted that the display node pre-stores a mapping relationship between various user-set alarm information and warning colors. For example, warning colors include red, blue, and yellow. Red can represent a collision, and blue can represent a following vehicle target. This embodiment of the application is not limited to this.

[0118] Taking the first target among multiple targets as an example, after the display node receives the warning information of the first target from the controller in the ADAS, it can search for the same warning information as the first target in the pre-stored mapping relationship between warning information and warning colors, and then use the warning color corresponding to the warning information as the warning color of the first target. The display node can then render the image model of the first target in the third road image to a color consistent with the warning color.

[0119] As can be seen from the introduction of the foregoing step 201, the marking information of each target includes an identification code. Based on this, the display node can search for the marking information containing the identification code of the first target in the third road image according to the identification code of the first target carried in the alarm information, and then determine the image model of the target corresponding to the marking information in the third road image as the image model corresponding to the first target. After that, the display node can render the image model of the first target in a color consistent with the warning color. According to the same method, the display node can add the alarm information of multiple targets in the third road image, and then use the third road image with the alarm information added as the second road image.

[0120] Step 203: Display the second road image.

[0121] After obtaining the second road image, the display node displays the second road image so that the user can control the vehicle based on the displayed second road image. For example, the displayed second road image can be as Figure 4 shown.

[0122] In the embodiment of the present application, the first road image sent by the controller in the ADAS includes the marking information of multiple targets detected by the controller. On this basis, the information of the targets included in the second road image obtained based on the marking information of multiple targets and the first road image will be more complete, reducing the probability of missing display or misdisplay of the road image, and thus being able to better assist the driver in driving the vehicle.

[0123] In addition, in the embodiment of the present application, the image models of the corresponding targets are superimposed on the image positions of the marking information of each target in the first road image. In this way, for the targets detected in the road image, the display node can accurately distinguish based on the marking information, and thus can achieve the accurate display of the targets in the final image.

[0124] Finally, in the embodiment of the present application, the second road image displayed by the display node may also include alarm information, so that the driver can control the vehicle based on the alarm information in the second road image displayed by the display node, and can largely avoid the occurrence of accidental accidents caused by the driver's negligence.

[0125] Next, the road image display device provided by the embodiment of the present application will be introduced.

[0126] See Figure 5 , the embodiment of the present application provides a road image display device 500, and the device 500 includes: a receiving module 501, an obtaining module 502, and a display module 503.

[0127] A receiving module 501, configured to receive a first road image sent by a controller in an ADAS, where the first road image includes marking information of multiple targets detected by the controller;

[0128] An obtaining module 502, configured to obtain a second road image based on the marking information of the multiple targets and the first road image;

[0129] A display module 503, configured to display the second road image.

[0130] Optionally, the obtaining module 502 is further configured to:

[0131] Overlay image models of the multiple targets on the first road image based on the marking information of the multiple targets to obtain a third road image;

[0132] Obtain the second road image based on the third road image.

[0133] Optionally, the obtaining module 502 is further configured to:

[0134] Crop the first road image based on a calibration size;

[0135] Obtain the marking information of each target in the cropped first road image;

[0136] Overlay the corresponding image model of the target on the cropped first road image based on the obtained marking information of each target.

[0137] Optionally, the marking information of each target includes the target type, image position, and angle of the corresponding target;

[0138] The obtaining module 502 is further configured to:

[0139] Obtain the corresponding image model of the target based on the target type of the first target, where the first target is any target in the cropped first road image;

[0140] Overlay the image model of the first target at the image position of the first target in the cropped first road image, and adjust the angle of the image model of the first target based on the angle of the first target.

[0141] Optionally, the obtaining module 502 is further configured to:

[0142] Remove background information from the third road image;

[0143] Add alarm information to the third road image after removing the background information to obtain the second road image.

[0144] In summary, in the embodiment of the present application, the first road image sent by the controller in the ADAS includes the marking information of multiple targets detected by the controller. On this basis, the information of the targets included in the second road image obtained based on the marking information of the multiple targets and the first road image will be more complete, reducing the probability of missing display or misdisplay of the road image, and thus being able to better assist the driver in driving the vehicle.

[0145] It should be noted that when the road image display device provided in the above embodiment displays the road image, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the road image display device provided in the above embodiment and the embodiment of the road image display method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0146] Figure 6 It is a block diagram of the structure of a display device 600 shown according to an exemplary embodiment. The display node in the above embodiment can be implemented through the display device 600.

[0147] Generally, the display device 600 includes a processor 601 and a memory 602.

[0148] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0149] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 is used to store at least one instruction, where the at least one instruction is used to be executed by the processor 601 to implement the road image display method provided in the method embodiments of the present application.

[0150] In some embodiments, the display device 600 may further include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602, and the peripheral device interface 603 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 603 through a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 604, a display screen 605, a positioning component 606, and a power supply 607.

[0151] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, and the present embodiment does not limit this.

[0152] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so on. The radio frequency circuit 604 may communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, each generation of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may further include a circuit related to NFC (Near Field Communication), and the present application does not limit this.

[0153] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on or above the surface of the display screen 605. The touch signals can be input to the processor 601 as control signals for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 605, which is disposed on the front panel of the display device 600; in other embodiments, there can be at least two display screens 605, which are respectively disposed on different surfaces of the display device 600 or are in a folded design; in some embodiments, the display screen 605 can be a flexible display screen, which is disposed on a curved surface or a folding surface of the display device 600. Even further, the display screen 605 can be set to an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 605 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode). It should be noted that in the embodiments of the present application, when the display device 600 is in a landscape orientation, the aspect ratio of the display screen of the display device 600 is greater than 1. For example, the aspect ratio of the display screen of the display device 600 can be 16:9 or 4:3. When the display device 600 is in a portrait orientation, the aspect ratio of the display screen of the display device 600 is less than 1. For example, the aspect ratio of the display screen of the display device 600 can be 9:18 or 3:4, etc.

[0154] The positioning component 606 is used to locate the current geographical location of the display device 600 to implement navigation or LBS (Location-Based Service). The positioning component 606 can be a positioning component based on the GPS (Global Positioning System), the Beidou system, or the Galileo system.

[0155] The power supply 607 is used to supply power to each component in the display device 600. The power supply 607 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When the power supply 607 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0156] In some embodiments, the display device 600 further includes one or more sensors.

[0157] That is to say, the embodiments of the present application not only provide a display device, including a processor and a memory for storing processor-executable instructions, wherein the processor is configured to execute Figure 2 the road image display method shown, but also the embodiments of the present application further provide a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by the processor, it can implement Figure 2 the road image display method shown.

[0158] Those skilled in the art can understand that Figure 6 the structure shown in does not constitute a limitation on the display device 600. The display device may include more or fewer components than shown in the figure, or combine some components, or adopt different component arrangements.

[0159] The embodiments of the present application further provide a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the terminal, the terminal can execute the road image display method provided in the above embodiments.

[0160] The embodiments of the present application further provide a computer program product containing instructions. When it runs on a computer, the computer is enabled to execute the Figure 2 road image display method provided in the embodiments shown.

[0161] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data), and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0162] The above description is not intended to limit the embodiments of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A method for displaying road images, characterized in that, The method includes: Receiving a first road image sent by a controller in an Advanced Driving Assistance System (ADAS), where the first road image includes marking information of multiple targets detected by the controller, and the marking information of each target includes the target type, image position, and angle of the corresponding target. Among them, the controller in the ADAS determines the angle of the corresponding target in the first road image for each target detection box based on the corner points of the detection results output by the road target detection model; Calibrating the valid area in the received first road image based on the calibration size, and cropping the part of the first road image other than the valid area based on the valid area to obtain a cropped first road image; Obtaining the marking information of each target in the cropped first road image; Based on the target type of the first target, obtaining an image model corresponding to the corresponding target, where the first target is any target in the cropped first road image; Overlaying the image model of the first target at the image position of the first target in the cropped first road image, and adjusting the angle of the image model of the first target based on the angle of the first target, so that the angle of the image model of the first target is consistent with the angle of the first target in the cropped first road image, to obtain a third road image; Removing the background information from the third road image; Adding alarm information to the third road image after removing the background information to obtain a second road image; Displaying the second road image.

2. A road image display device, characterized in that, The device includes: A receiving module, configured to receive a first road image sent by a controller in an Advanced Driving Assistance System (ADAS), where the first road image includes marking information of multiple targets detected by the controller, and the marking information of each target includes the target type, image position, and angle of the corresponding target. Among them, the controller in the ADAS determines the angle of the corresponding target in the first road image for each target detection box based on the corner points of the detection results output by the road target detection model; A cropping module, configured to calibrate the valid area in the received first road image based on the calibration size, and crop the part of the first road image other than the valid area based on the valid area to obtain a cropped first road image; A first obtaining module, configured to obtain the marking information of each target in the cropped first road image; A second obtaining module, configured to obtain an image model corresponding to the corresponding target based on the target type of the first target, where the first target is any target in the cropped first road image; A covering module, configured to cover an image model of the first target at the image position of the first target in the cropped first road image, and adjust the angle of the image model of the first target based on the angle of the first target, so that the angle of the image model of the first target is consistent with the angle of the first target in the cropped first road image, thereby obtaining a third road image; A removing module, configured to remove background information in the third road image; An adding module, configured to add alarm information to the third road image after removing the background information, thereby obtaining a second road image; A displaying module, configured to display the second road image.

3. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the road image display method described in claim 1 is implemented.

Images