Lane line display method, mobile device and lane line display device

By acquiring and detecting lane line information on mobile devices and overlaying lane line interface elements onto the displayed image, the problem of visual element overload is solved, improving the operator's judgment efficiency and driving safety.

CN123111318APending Publication Date: 2026-11-17GUANGZHOU XIAOPENG CONNECTIVITY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611543073.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-30
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

When mobile devices display traffic information, the visual elements are too dense, which reduces the efficiency and accuracy of operators' judgment and affects operational safety.

Method used

By acquiring images from multiple image acquisition devices, lane lines are detected to determine their position and direction. Lane line interface elements, including lane line labels and direction elements, are then overlaid on the displayed image to simplify information presentation.

Benefits of technology

It improves the simplicity of mobile device interface display, reduces distraction, and enhances the operator's efficiency in identifying lane information and driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123111318A_ABST
    Figure CN123111318A_ABST
Patent Text Reader

Abstract

This application discloses a lane line display method, a mobile device, and a lane line display apparatus. The method includes: acquiring first images respectively captured by multiple image acquisition devices on the mobile device; performing lane line detection on each first image to determine one or more lane lines contained in each first image; determining lane line interface elements to be superimposed and displayed in the image to be displayed based on the one or more lane lines contained in the image to be displayed; and displaying a target image corresponding to each image to be displayed based on each lane line interface element. The lane line interface elements are used to represent the position and / or type of the corresponding lane line, and to represent the road direction of the corresponding lane line. This application can improve the simplicity of the mobile device interface display and reduce the risk of operator distraction due to repeated observation of surrounding road conditions, effectively improving the operational safety of the mobile device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of interface display technology, specifically to a lane line display method, a mobile device, and a lane line display apparatus. Background Technology

[0002] Currently, relevant technologies typically identify and analyze the content of a forward-facing view on a mobile device, then overlay the analyzed road condition information onto the screen in a visual manner. This allows operators to accurately and intuitively obtain road condition information, assisting them in controlling the mobile device. However, due to the large variety and quantity of road condition information, mobile devices not only overlay basic road elements such as traffic signs and lane lines on the screen, but also often additionally overlay directional guidance elements such as guide arrows. This can easily lead to an overly dense visual element display, affecting the efficiency and accuracy of the operator's judgment of the overall road conditions and reducing the operational safety of mobile devices in complex traffic environments. Summary of the Invention

[0003] This application discloses a lane line display method, a mobile device, and a lane line display apparatus, which can improve the simplicity of the mobile device interface display and reduce the risk of operator distraction due to repeated observation of surrounding road conditions, effectively improving the operational safety of the mobile device.

[0004] The first aspect of this application discloses a lane line display method applied to a mobile device, the method comprising: Acquire the first image captured by multiple image acquisition devices on the mobile device; Lane line detection is performed on each of the first images to determine one or more lane lines contained in each of the first images; Based on one or more lane lines contained in the image to be displayed, determine the lane line interface element to be superimposed and displayed in the image to be displayed, wherein the image to be displayed is one or more of the first images. Based on each of the lane line interface elements, display the target image corresponding to each of the images to be displayed; The target image is an image on which lane line interface elements corresponding to each lane line are superimposed on an image to be displayed. The lane line interface elements are used to represent the position and / or type of the corresponding lane line, and to represent the road direction of the corresponding lane line.

[0005] In some possible embodiments, the lane line interface elements include lane line identification elements and lane line direction elements. The lane line identification elements are used to characterize the position and lane line type of the corresponding lane line in the image to be displayed, and the lane line direction elements are used to characterize the road direction of the lane line.

[0006] In some possible embodiments, the color of the lane line identifier element is the same as the color of the corresponding lane line, and the lane line type of the lane line identifier element matches the lane line type of the corresponding lane line. or, The color of the lane line marking element includes a first color and a second color. The first color is used to represent the first lane line type of the lane line corresponding to the lane line marking element, and the second color is used to represent the second lane line type of the lane line corresponding to the lane line marking element.

[0007] In some possible embodiments, the lane line direction element includes a plurality of pointing elements, which are arranged sequentially at intervals along the road direction of the corresponding lane line on the corresponding image to be displayed.

[0008] In some possible embodiments, the spacing between the plurality of pointing elements is determined based on the curvature of the lane line corresponding to the lane line identifier element.

[0009] In some possible embodiments, displaying the target image corresponding to each of the images to be displayed includes: At the positions corresponding to each lane line contained in each of the images to be displayed, the lane line interface elements corresponding to each lane line are overlaid and displayed.

[0010] In some possible embodiments, the step of performing lane line detection on each of the first images to determine that each of the first images contains one or more lane lines includes: Lane line detection is performed on each of the first images to determine the image position information and direction information corresponding to one or more lane lines contained in each of the first images. The step of determining the lane line interface element to be superimposed and displayed based on one or more lane lines contained in the image to be displayed includes: Based on the image position and direction information corresponding to each lane line in the image to be displayed, a lane line interface element is generated to be superimposed and displayed on each lane line in the image to be displayed.

[0011] In some possible embodiments, the step of performing lane line detection on each of the first images to determine the image position information and direction information corresponding to one or more lane lines contained in each of the first images includes: The first images are stitched together to obtain a bird's-eye view; Lane line detection is performed on the bird's-eye view to obtain the set of pixels and direction information corresponding to one or more lane lines contained in the bird's-eye view; Based on the conversion relationship between the bird's-eye view and each of the first images, the set of pixels corresponding to each lane line is mapped to each of the first images to obtain the image position information and direction information corresponding to each lane line contained in each of the first images.

[0012] In some possible embodiments, the step of performing lane line detection on the bird's-eye view to obtain the set of pixels and direction information corresponding to one or more lane lines contained in the bird's-eye view includes: The bird's-eye view is input into the lane detection model, and the model extracts features from the bird's-eye view to obtain the set of pixels corresponding to one or more lane lines in the bird's-eye view, as well as the direction information corresponding to each lane line.

[0013] In some possible embodiments, the direction information includes the road direction corresponding to the lane line and the extension direction corresponding to the lane line; the step of inputting the bird's-eye view into the lane line detection model, and extracting features from the bird's-eye view through the lane line detection model to obtain the set of pixels corresponding to one or more lane lines contained in the bird's-eye view, and the direction information corresponding to each lane line, includes: The bird's-eye view is input into the lane line detection model, and the lane line detection model performs feature extraction on the bird's-eye view to obtain a lane line heatmap and a pixel offset map. The pixel value of each pixel in the lane line heatmap is used to represent the probability that the pixel belongs to the lane line, and the pixel offset map is used to represent the pixel offset between adjacent pixels in the lane line heatmap. Based on the lane heatmap, lane lines are extracted from each lane line in the bird's-eye view to obtain the set of pixels corresponding to each lane line. Based on the set of pixels corresponding to each lane line, the road direction corresponding to each lane line is determined; Based on the set of pixels corresponding to each lane line, obtain the pixel offset corresponding to each set of pixels in the pixel offset map, and determine the lane line offset corresponding to each set of pixels based on the pixel offset corresponding to each set of pixels. Based on the lane offset of each lane line contained in the bird's-eye view, the extension direction of each lane line is determined.

[0014] In some possible embodiments, the step of extracting lane lines from each lane line included in the bird's-eye view based on the lane line heatmap to obtain a set of pixels corresponding to each lane line includes: Based on a preset probability threshold, the lane heatmap is binarized to obtain a lane foreground image. Identify one or more connected regions contained in the lane line foreground image; Target processing operations are performed on the one or more connected regions to obtain a set of pixels corresponding to one or more lane lines. The target processing operations include one or more of the following: skeleton thinning, pseudo-branch and branch removal, path sorting, and resampling.

[0015] In some possible embodiments, the step of obtaining the pixel offset corresponding to each pixel set in the pixel offset map based on the pixel set corresponding to each lane line, and determining the lane line offset corresponding to each pixel set based on the pixel offset corresponding to each pixel set, includes: Based on the set of target pixels corresponding to the target lane line in the bird's-eye view, the pixel offsets corresponding to the multiple target pixels included in the target pixel set are determined in the pixel offset map, and the target lane line is any of the lane lines. The sum of the pixel offsets corresponding to the plurality of target pixels is calculated to obtain the cumulative offset between the first target pixel and the second target pixel in the target pixel set, which is used as the lane line offset corresponding to the target lane line. The first target pixel is the first target pixel in the target pixel set, and the second target pixel is the last target pixel in the target pixel set.

[0016] In some possible embodiments, the pixel offset map includes a first pixel offset map and a second pixel offset map. The first pixel offset map is used to characterize the pixel offset between adjacent pixels in the lane line heatmap on a first coordinate axis, and the second pixel offset map is used to characterize the pixel offset between adjacent pixels in the lane line heatmap on a second coordinate axis. The first coordinate axis is perpendicular to the second coordinate axis. The accumulated offset includes a first accumulated offset corresponding to the first coordinate axis and a second accumulated offset corresponding to the second coordinate axis; The step of determining the extension direction of each lane line based on the lane line offset of each lane line included in the bird's-eye view includes: Based on the first target pixel, the second target pixel, the first accumulated offset, and the second accumulated offset, the target arrangement order of multiple target pixels in the target pixel set is determined, and the direction formed by connecting multiple target pixels in the target pixel set sequentially along the target arrangement order is used as the extension direction corresponding to the target lane line.

[0017] In some possible embodiments, the target arrangement order includes a first arrangement order or a second arrangement order, wherein the first arrangement order is the order in which multiple target pixels in the target pixel set are arranged in a forward direction, and the second arrangement order is the order in which multiple target pixels in the target pixel set are arranged in a reverse direction; the multiple target pixels in the target pixel set are arranged sequentially along the first arrangement order; Determining the target arrangement order of multiple target pixels in the target pixel set based on the first target pixel, the second target pixel, the first accumulated offset, and the second accumulated offset includes: The first offset coordinate is determined based on the first pixel coordinates corresponding to the first target pixel, the first accumulated offset, and the second accumulated offset. The second offset coordinate is determined based on the second pixel coordinates corresponding to the second target pixel, the first accumulated offset, and the second accumulated offset. Calculate the first distance difference between the first pixel coordinates and the second offset coordinates, and calculate the second distance difference between the second pixel coordinates and the first offset coordinates; If the second distance difference is less than or equal to the first distance difference, then the target arrangement order is determined to be the first arrangement order; If the second distance difference is greater than the first distance difference, then the target arrangement order is determined to be the second arrangement order.

[0018] A second aspect of this application discloses a mobile device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any of the above embodiments.

[0019] A third aspect of this application discloses a lane line display device, including a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any of the above embodiments.

[0020] This application discloses a lane line display method, a mobile device, and a lane line display apparatus. The mobile device acquires first images captured by multiple image acquisition devices on the mobile device, performs lane line detection on each first image, and determines one or more lane lines contained in each first image. Based on the one or more lane lines contained in the image to be displayed, the mobile device determines lane line interface elements to be superimposed and displayed on the one or more lane lines in the image to be displayed. The image to be displayed is one or more of the first images. Based on each lane line interface element, the mobile device displays a target image corresponding to each image to be displayed. A target image is an image on which the lane line interface elements corresponding to each lane line are superimposed on an image to be displayed. The lane line interface elements are used to represent the position and / or type of the corresponding lane line, and to represent the road direction of the corresponding lane line.

[0021] Through the above implementation method, traffic information such as lane type and road direction can be directly superimposed on the lane lines in the image to be displayed, without the need to display traffic information separately outside the lane lines. This avoids information overload caused by displaying a large amount of additional traffic information (such as directional arrows) in complex traffic scenarios such as multiple intersections or multiple lanes, improving the simplicity of the mobile device interface and the efficiency of information acquisition. Furthermore, by displaying the target image corresponding to each image to be displayed, the operator can directly obtain traffic information such as lane direction from any viewpoint of the display screen, reducing the risk of lane misjudgment and deviation from the surrounding roads of the mobile device. This reduces the distraction caused by the operator repeatedly observing the surrounding road conditions, effectively improving the operational safety of the mobile device. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1A This is a schematic diagram of the structure of a mobile device provided in an embodiment of this application; Figure 1B A schematic diagram of a camera module provided in an embodiment of this application; Figure 1C A schematic diagram showing road images in various directions provided in an embodiment of this application; Figure 2 A flowchart illustrating the lane line display method provided in this application embodiment; Figure 3A A schematic diagram of a lane line interface element provided in an embodiment of this application; Figure 3B A schematic diagram of another lane line interface element provided in an embodiment of this application; Figure 4A A schematic diagram of a pointing element in a lane provided in an embodiment of this application; Figure 4B A schematic diagram of another lane pointing element provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the determination of the spacing between pointing elements based on curvature, provided for an embodiment of this application; Figure 6A A schematic diagram of a target image provided for an embodiment of this application; Figure 6B A schematic diagram of multiple target images provided in an embodiment of this application; Figure 7 A flowchart for determining the image position information and direction information corresponding to lane lines provided in an embodiment of this application; Figure 8 A flowchart for detecting lane lines on a bird's-eye view provided in this application embodiment; Figure 9 A flowchart for determining the set of pixels corresponding to each lane line provided in this application embodiment; Figure 10 A flowchart for determining the target arrangement order provided in this application embodiment; Figure 11A A schematic diagram illustrating the determination of the target arrangement order provided in an embodiment of this application; Figure 11B A schematic diagram illustrating another method for determining the arrangement order of targets, provided in an embodiment of this application; Figure 12 This is a structural block diagram of a mobile device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0026] This lane line display method can be applied to mobile devices, which may include motor vehicles such as cars, non-motor vehicles such as electric vehicles and bicycles with driving recorders, or various devices with autonomous mobility such as unmanned transport vehicles. Mobile devices may also include devices that are not movable themselves but can move with the operator or carrier, such as head-mounted augmented reality devices, but are not limited to these.

[0027] Figure 1A This is a schematic diagram of the structure of a mobile device provided in an embodiment of this application. Figure 1A As shown, the mobile device 100 may include a control module 101, a camera module 102, and a display module 103. The control module 101 is connected to the camera module 102 and the display module 103. The control module 101 can control the camera module 102 to acquire road images of the mobile device 100 in one or more directions, and control the display module 103 to display road images in one or more directions.

[0028] Road images can refer to images that include one or more elements such as lane lines, road markings, traffic signs, and surrounding environmental elements, primarily focusing on ground roads.

[0029] In some embodiments, the control module 101 can analyze road images in each direction to obtain road condition information in each direction, and overlay the road condition information in each direction when the control display module 103 displays road images in each direction.

[0030] Traffic information may include, but is not limited to, one or more of the following: lane position, lane type, road direction, obstacle distribution, traffic sign content, and road surface condition.

[0031] In some embodiments, the camera module 102 may include multiple image acquisition devices, each image sensor may correspond to one direction of the mobile device 100, and the control module 101 may control each image sensor to acquire road images in the corresponding direction.

[0032] Optionally, the camera module 102 may include only one image acquisition device, which can be used to acquire images of the mobile device 100 in multiple directions. For example, the image acquisition device is rotatable, and the control module 101 can control the image acquisition device to rotate, and acquire images in the corresponding direction after rotating to a preset angle, thereby obtaining images in various directions.

[0033] In some embodiments, the display module 103 may be a display inside the mobile device 100.

[0034] Furthermore, if the mobile device 100 is a car, the display module 103 can also be the windshield of the car, and the control module 101 can project one or more road images captured by the camera module 102 onto the windshield of the car through head-up display technology.

[0035] Optionally, the display module 103 may also be a display outside the mobile device 100, and the control module 101 is communicatively connected to the display module 103. The control module 101 may send one or more road images captured by the camera module 102 to the display module 103, and the display module 103 may display the received one or more road images.

[0036] Figure 1B This is a schematic diagram of a camera module provided in an embodiment of this application. Figure 1B As shown, in some embodiments, the mobile device 100 may be a car, and the car's camera module 102 may include six image acquisition units, from image acquisition unit 1021 to image acquisition unit 1026, with each image acquisition unit facing a different direction of the car to acquire road images in that direction. For example, image acquisition unit 1021 is used to acquire road images directly in front of the car, image acquisition unit 1022 is used to acquire road images to the right front of the car, image acquisition unit 1023 is used to acquire road images to the left front, image acquisition unit 1024 is used to acquire road images to the right rear, image acquisition unit 1025 is used to acquire road images to the left rear, and image acquisition unit 1026 is used to acquire road images to the rear. The control module 101 can control the display module 103 to display road images in one or more directions.

[0037] Figure 1C This is a schematic diagram showing road images in various directions, provided as an embodiment of this application. For example... Figure 1C As shown, the control module 101 can control the display module 103 to simultaneously display the road images collected by the image acquisition devices 1021, 1022, 1023 and 1026, so that the operator can observe the road conditions in four directions at the same time.

[0038] In other embodiments, the mobile device 100 may also be a head-mounted augmented reality device. When an operator wears the head-mounted augmented reality device, the device can capture road images in real time from various directions via the camera module 102, and overlay the road images and corresponding road condition information in each direction onto the operator's real field of vision in an augmented reality manner. Furthermore, when the operator turns their head to look in a certain direction, the device can automatically display the road image and road condition information corresponding to that direction, thereby helping the operator quickly perceive the lane structure and direction to plan a driving route more safely.

[0039] Currently, in related technologies, when the mobile device 100 displays a view from the front (i.e., only displays road images captured by the image acquisition device 1021), the content of the image is identified and analyzed, and the analyzed road condition information is overlaid on the display screen in a visual manner. This allows the operator to accurately and intuitively obtain the road condition information from that viewpoint, assisting the operator in controlling the mobile device 100. For example, when the mobile device 100 is a car, the display screen can show the view directly in front of the car while the car is moving forward, and overlay the road condition information directly in front of the car on the display screen to guide the operator to control the car to drive in the correct lane.

[0040] However, when road condition information is overlaid on the display screen in a visual manner, the large variety and quantity of road condition information, in addition to basic road elements such as traffic signs, traffic lights, and lane lines, often includes additional functional elements such as directional arrows, turn prompts, and route guidance. This results in an overly dense visual element display, easily leading to information overload and visual interference, and increasing the information identification cost for operators. Furthermore, when mobile devices display multiple perspectives, they typically only provide basic raw image feedback. The effective road condition information directly readable on the display screen is severely lacking, requiring operators to manually analyze and obtain road condition information from the raw images. This affects the efficiency and accuracy of operators' judgment of the surrounding road conditions of the mobile device 100, reducing the operational safety of the mobile device 100 in complex traffic environments.

[0041] Figure 2 A flowchart illustrating the lane line display method provided in this application embodiment. Figure 2 As shown, the method may include the following steps: Step 202: Acquire the first image by multiple image acquisition devices on the mobile device.

[0042] Multiple first images are road images captured simultaneously by multiple image capture devices on a mobile device.

[0043] In some embodiments, the image acquisition device can continuously acquire road images at a preset frame rate, and the mobile device can acquire the first image corresponding to the current video frame transmitted by multiple image acquisition devices.

[0044] Step 204: Perform lane line detection on each first image to determine one or more lane lines contained in each first image.

[0045] In one implementation, the mobile device can perform lane line detection on each first image using a lane line detection model to determine one or more lane lines contained in each first image.

[0046] Lane detection models may include, but are not limited to, color thresholding-based models, edge detection combined with Hough transform models, RANSAC-based models, or deep learning models such as LaneNet.

[0047] As another implementation, the mobile device can merge the first images into a bird's-eye view, and use a lane line detection model to detect lane lines in the bird's-eye view to determine one or more lane lines contained in the bird's-eye view. Based on the one or more lane lines contained in the bird's-eye view, one or more lane lines contained in each first image can be determined.

[0048] Step 206: Based on one or more lane lines contained in the image to be displayed, determine the lane line interface element to be superimposed and displayed on one or more lane lines in the image to be displayed.

[0049] The image to be displayed is one or more of the first images. The image to be displayed may refer to the road image that the mobile device determines to be displayed based on the current operating functions of the mobile device or the current operation of the user.

[0050] For example, if the mobile device is a car and the car's parking assist function is currently activated, the car's control module can determine that the image to be displayed is the road image captured by the image acquisition device directly behind the vehicle. If the car is currently changing lanes or turning, such as turning right, the driver needs to observe oncoming vehicles and blind spots from the side and rear, so the control module can determine that the image to be displayed includes road images captured by image acquisition devices from three directions: right, directly behind, and right rear. If the car detects that the user manually selects image acquisition devices for each direction via the touchscreen, the control module can determine that the image to be displayed includes road images captured by each image acquisition device manually selected by the user.

[0051] In some embodiments, a mobile device can generate lane line interface elements that will be superimposed and displayed on each lane line in the image to be displayed, based on the image position information and direction information corresponding to each lane line in the image to be displayed.

[0052] The image location information includes the set of pixels corresponding to each lane line, and the set of pixels corresponding to each lane line is determined based on multiple pixels in the image to be displayed whose probability of belonging to a lane line is greater than a preset probability threshold.

[0053] The direction information corresponding to each lane line is determined based on the semantic information of the set of pixels corresponding to each lane line and the target arrangement order of multiple pixels in the set of pixels. The semantic information includes color and / or lane line type.

[0054] In some embodiments, the mobile device can determine the color and lane line type of the lane line interface element corresponding to each lane line based on the color and lane line type of one or more lane lines contained in the image to be displayed.

[0055] Lane line interface elements are used to represent the location and / or type of the corresponding lane line, as well as the road direction of the corresponding lane line. The lane line type can indicate the boundaries of vehicle travel and whether crossing is permitted. The road direction refers to the direction of travel that vehicles are allowed to take within the lane.

[0056] In some embodiments, lane line interface elements may be composed of multiple pointer elements connected in sequence to simultaneously represent the lane line type and road direction of the corresponding lane line.

[0057] Figure 3A This is a schematic diagram of a lane line interface element provided in an embodiment of this application. Figure 3A As shown, lane line interface elements can be composed of multiple directional elements 301 "^". For lane lines of the dashed type, each segment of the dashed line is composed of multiple directional elements connected sequentially, and there are no directional elements in the blank spaces between two segments, thus making the lane line interface elements form a visual form of dashed line segments and discontinuities. For lane lines of the solid type, multiple directional elements are arranged continuously and without interruption along the road direction, without any blank interruption areas, to represent solid lane lines.

[0058] The above implementation method can achieve integrated presentation of lane line type and road direction, making the lane line type and traffic guidance meaning more intuitive and concise, which helps to improve the operator's comprehensive identification efficiency of lane information.

[0059] In some embodiments, the lane line interface elements include lane line identification elements and lane line direction elements. The lane line identification elements are used to represent the position and lane line type of the corresponding lane line in the image to be displayed, and the lane line direction elements are used to represent the road direction of the lane line.

[0060] In some embodiments, the mobile device can determine the color and lane line type of the lane line identifier element corresponding to each lane line based on the color and lane line type of one or more lane lines contained in the image to be displayed.

[0061] In one implementation, the color of the lane line marker element is the same as the color of the corresponding lane line, and the lane line type of the lane line marker element matches the lane line type of the corresponding lane line.

[0062] For example, when the lane line type is double solid yellow lines, the corresponding lane line indicator element is yellow, and the lane line type is solid; when the lane line type is single dashed white lines, the corresponding lane line indicator element is white, and the lane line type is dashed. Furthermore, the lane line indicator element can be a line style with a preset width, or it can be a strip area style that matches the width of the corresponding lane line, but it is not limited to these.

[0063] Furthermore, when the lane line is a special lane line with a center line or baseline, the color of the corresponding lane line identifier element can be the color corresponding to the center line or baseline, and the lane line type of the corresponding lane line identifier element can be the same as the lane line type corresponding to the center line or baseline. For example, when the lane line type is a herringbone pattern with a dashed center line, the color of the corresponding lane line identifier element can be the color of the dashed center line of the herringbone pattern, and the lane line type is dashed. When the lane line type is a zigzag pattern, the color of the corresponding lane line identifier element can be the color of the zigzag baseline, and the lane line type is solid.

[0064] The above implementation method ensures that the color and lane line type of the lane line marking elements accurately match each lane line in the image to be displayed, making the information expression of the lane lines more intuitive and consistent, thereby improving the operator's efficiency in identifying lane line types and driving safety.

[0065] In another implementation, the color of the lane line marking element includes a first color and a second color. The first color is used to represent the first lane line type of the lane line corresponding to the lane line marking element, and the second color is used to represent the second lane line type of the lane line corresponding to the lane line marking element.

[0066] For example, the first lane line type can be a solid line, the second lane line type can be a dashed line, the color corresponding to the first lane line type can be red, and the color corresponding to the second lane line type can be green.

[0067] Through the above implementation method, the mobile device can determine the first and second colors in the corresponding lane line identification elements to distinguish different types such as solid lines and dashed lines according to the lane line type of each lane line. This allows operators to intuitively judge the solid and dashed attributes of the lane lines simply by their colors without having to carefully identify the line styles, thereby reducing the identification burden and further improving information transmission efficiency and the operational safety of the mobile device.

[0068] In some embodiments, the lane line direction element includes multiple pointing elements, which are arranged sequentially and at intervals along the road direction of the corresponding lane line on the corresponding image to be displayed.

[0069] Pointer elements can be used to indicate the road direction of lane lines, i.e., lane direction. Pointer elements can be in the form of arrows or triangles, among other things.

[0070] In one implementation, multiple pointing elements are arranged sequentially and at intervals along the road direction of the corresponding lane line on the corresponding image to be displayed, according to the lane line type of the corresponding lane line.

[0071] For example, when the lane line type is a solid line, multiple pointing elements are arranged at equal intervals along the road direction of the solid line in the corresponding image to be displayed; when the lane line type is a dashed line, multiple pointing elements can be arranged along the road direction of the dashed line in the corresponding image to be displayed, according to the line segments and intervals of the dashed line.

[0072] Figure 3B This is a schematic diagram of another lane line interface element provided in an embodiment of this application. For example... Figure 3B As shown, the pointing element is arrow 302. When the lane line type is a solid line, the lane line identifier element 303 is a solid line, and multiple arrows 302 are arranged at equal intervals along the road direction corresponding to the lane line identifier element 303. When the lane line type is a dashed line, the lane line identifier element 304 is a dashed line, and multiple arrows 302 are arranged along the road direction corresponding to the lane line identifier element 304 according to the line segment and interval of the lane line identifier element 304, with one arrow 302 arranged for each line segment.

[0073] The color of the pointing element can be the same as the color of the lane line marking element. It should be noted that... Figure 3B The fact that the color of the middle arrow 302 is different from that of the lane line marking element 303 is only used to illustrate the shape of the arrow 302, and is not used to indicate that the color of the pointing element is different from that of the lane line marking element.

[0074] Optionally, the color of the pointing element can also be different from the color of the lane marking element to more clearly indicate the road direction of the lane.

[0075] Through the above implementation method, multiple pointing elements are arranged sequentially and at intervals along the road direction of the corresponding lane lines on the corresponding image to be displayed. This not only eliminates the need to occupy additional display space and reduces the cognitive load on operators, but also improves the accuracy and intuitiveness of the road direction information of the lane lines, further ensuring driving safety.

[0076] In some embodiments, the direction indicated by the pointing element is related to the lane line type in addition to the road direction.

[0077] When the lane line type is a single line, if the road directions on both sides of the lane line are in the same direction, the direction indicated by the pointing element is the same as the road direction. If the road directions on both sides of the lane line are opposite, the pointing element includes a first sub-element in the same direction as the first road direction and a second sub-element in the same direction as the second road direction. The first road direction and the second road direction are opposite.

[0078] When the lane line type is double-line, if the road directions on both sides of the lane line are in the same direction, the direction indicated by the pointing element is the same as the road direction. If the road directions on both sides of the lane line are opposite, then in the two lane lines of the double-line type, the pointing element corresponding to the lane line closer to the first road direction indicates the same direction as the first road direction, and the pointing element corresponding to the lane line closer to the second road direction indicates the same direction as the second road direction.

[0079] Figure 4A This is a schematic diagram of a lane pointing element provided in an embodiment of this application. For example... Figure 4A As shown in (a), the first road direction X is Figure 4A The upward direction, the second road direction Y is Figure 4A In the downward direction, the gray area in the diagram represents lane lines. When lane line 400 is the middle lane of a two-lane road system, and the road directions on either side of lane line 400 are opposite, the pointing element 401 corresponding to lane line 400 includes a first sub-element 4011 in the same direction as the first road direction X and a second sub-element 4012 in the same direction as the second road direction Y, used to simultaneously indicate both the first road direction X and the second road direction Y; for example... Figure 4A As shown in (b), when lane lines 402 and 403, and lane lines 404 and 405 are all double lines (e.g., double yellow lines), the road directions on both sides of lane lines 402 and 403 are opposite. Therefore, the directional element corresponding to lane line 402 points to the first road direction X, and the directional element corresponding to lane line 403 points to the second road direction Y. On the other hand, the road directions on both sides of lane lines 404 and 405 are the same. Therefore, the directional elements corresponding to lane lines 404 and 405 both point to the first road direction X.

[0080] In some embodiments, the direction indicated by the pointing element is also related to the lane type, wherein the mobile device may determine the lane type based on the directional arrows on the lane, but is not limited thereto.

[0081] When the lane type is a one-way single lane, the directional elements corresponding to the lane lines on both sides of the lane indicate the same direction as the road direction. When the lane type is a two-way single lane, the directional elements corresponding to the lane lines on the right side of the lane indicate the same direction as the movement direction of the mobile device, while the directional elements corresponding to the lane lines on the left side of the lane indicate the opposite direction, to indicate that the mobile device is moving to the right side of the road. If the mobile device is a vehicle, and the vehicle is a right-hand drive vehicle, then the directional elements corresponding to the lane lines on the left side of the lane indicate the same direction as the movement direction of the mobile device, while the directional elements corresponding to the lane lines on the right side of the lane indicate the opposite direction, to indicate that the mobile device is moving to the left side of the road.

[0082] Figure 4B This is a schematic diagram of another lane pointing element provided in an embodiment of this application. For example... Figure 4B As shown in (a), when the lane type is one-way single lane and the mobile device 406 moves along the first road direction X, the directions indicated by the pointing elements corresponding to lane lines 407 and 408 are both in the same direction as the first road direction X, such as... Figure 4B As shown in (b), when the lane type is a two-way single lane and the mobile device 406 moves along the first road direction X, the direction indicated by the pointing element corresponding to the lane line 410 on the right side of the mobile device 406 is the first road direction X, and the direction indicated by the pointing element corresponding to the lane line 409 on the left side of the mobile device 406 is the second road direction Y.

[0083] In some embodiments, the spacing between multiple pointing elements is determined based on the curvature of the lane line corresponding to the lane line identifier element.

[0084] The spacing between multiple directional elements is inversely proportional to the curvature of the lane line corresponding to the lane line indicator element. The greater the curvature of the lane line corresponding to the lane line indicator element, the smaller the spacing between the multiple directional elements. When the curvature of the lane line is the same and not zero at all points, the multiple directional elements are arranged at equal intervals according to the second interval corresponding to the curvature. When the curvature of the lane line corresponding to the lane line indicator element is zero, the multiple directional elements are arranged at equal intervals according to the preset first interval, and the second interval corresponding to the curvature is less than the first interval.

[0085] Figure 5 This is a schematic diagram illustrating the determination of the spacing between pointing elements based on curvature, as provided in an embodiment of this application. Figure 5As shown, on a straight road, the curvature of the lane lines is 0 at all points. The multiple directional elements contained in the lane line marking elements are arranged at equal intervals according to a preset first interval distance, and the distribution of the directional elements is relatively sparse. However, on a curve, the curvature of the lane lines is the same at all points. Therefore, the multiple directional elements contained in the lane line marking elements are arranged at equal intervals according to a second interval distance corresponding to the curvature, and the distribution of the directional elements is relatively dense.

[0086] Understandably, lane markings on actual roads are not always straight; they often have curved sections depending on the road alignment design, resulting in curves with varying curvatures in some sections. When lane markings are curved, if multiple directional elements are still arranged at equal intervals, the directional elements may become sparse and the directional guidance discontinuous in areas of greater curvature. Therefore, by determining the spacing between multiple directional elements based on the curvature of the lane line corresponding to each lane marking element—the greater the curvature of the lane line corresponding to each lane marking element, i.e., the more curved the lane line—the denser the directional elements should be, thus creating a more cohesive directional guide. Compared to arranging multiple directional elements at equal intervals, dynamically adjusting the spacing between directional elements based on the curvature of the lane line can more smoothly and continuously outline the curve trajectory, improving the operator's ability to predict changes in road direction and enhancing visual continuity.

[0087] In some embodiments, the size of the pointing element is determined based on the positional distance between the lane line where the pointing element is located and the mobile device.

[0088] The size of the pointing element is directly proportional to its positional distance; the farther the distance, the smaller the size of the pointing element, in order to conform to the visual principle of "nearer is larger, farther is smaller" in actual observation. Optionally, multiple pointing elements included in the lane line direction element can also be of the same size.

[0089] Step 208: Based on each lane line interface element, display the target image corresponding to each image to be displayed. A target image is an image on which the lane line interface elements corresponding to each lane line are superimposed on an image to be displayed.

[0090] In some embodiments, the mobile device may overlay lane line interface elements corresponding to each lane line at the position corresponding to each lane line contained in each image to be displayed.

[0091] Figure 6A This is a schematic diagram of a target image provided for an embodiment of this application. For example... Figure 6AAs shown, the mobile device can display a target image 600, which is a superimposed display of lane line identification elements 601 and lane line direction elements corresponding to each lane line contained in the corresponding image to be displayed. The lane line direction elements include multiple pointing elements 602, which are in the form of arrows. The multiple pointing elements 602 are arranged sequentially and at intervals along the road direction of the lane line corresponding to the lane line identification element 601. The closer the pointing element 602 is to the mobile device, the larger the size of the corresponding pointing element 602.

[0092] Figure 6B This is a schematic diagram of multiple target images provided in an embodiment of this application. For example... Figure 6B As shown, the mobile device can simultaneously display target images corresponding to the front, right front, left front, and rear of the mobile device on the display module.

[0093] Through the above implementation method, traffic information such as lane type and road direction can be directly superimposed on the lane lines in the image to be displayed, without the need to display traffic information separately outside the lane lines. This avoids information overload caused by displaying a large amount of additional traffic information (such as directional arrows) in complex traffic scenarios such as multiple intersections or multiple lanes, improving the simplicity of the mobile device interface and the efficiency of information acquisition. Furthermore, by displaying the target image corresponding to each image to be displayed, the operator can directly obtain traffic information such as lane direction from any viewpoint of the display screen, reducing the risk of lane misjudgment and deviation from the surrounding roads of the mobile device. This reduces the distraction caused by the operator repeatedly observing the surrounding road conditions, effectively improving the operational safety of the mobile device.

[0094] In some embodiments, the mobile device can perform lane line detection on each first image, determine the image position information and direction information corresponding to one or more lane lines contained in each first image, and generate lane line interface elements to be superimposed and displayed on each lane line in the image to be displayed based on the image position information and direction information corresponding to each lane line in the image to be displayed.

[0095] The image location information corresponding to the lane line includes the pixel coordinates of multiple pixels in the set of pixels corresponding to the lane line.

[0096] The directional information corresponding to lane lines can include the road direction corresponding to the lane line and the direction in which the lane line extends. The direction in which a lane line extends can refer to the direction in which multiple pixels in the set of pixels corresponding to the lane line are connected sequentially.

[0097] In some embodiments, the mobile device can perform lane line detection on the bird's-eye view stitched together from the first images to determine the image position information and direction information corresponding to each lane line. The image position information corresponding to each lane line includes the set of pixels corresponding to each lane line. Figure 7 This is a flowchart illustrating the determination of image position and direction information corresponding to lane lines, provided in an embodiment of this application. (For example...) Figure 7 As shown, the step of detecting lane lines in each first image to determine the image position and direction information corresponding to one or more lane lines in each first image may include the following steps: Step 701: Stitch together the first images to obtain a bird's-eye view.

[0098] In some embodiments, the mobile device can input each first image into the LSS (Lift-Splat-Shoot) model, and perform viewpoint transformation and feature projection processing on each first image through the LSS model to obtain a bird's-eye-view (BEV).

[0099] Mobile devices can input each first image into an image backbone network, which extracts features from each first image to obtain feature data corresponding to each first image. Then, the feature data corresponding to each first image is input into an LSS model, which performs viewpoint transformation and feature projection processing on the feature data corresponding to each first image to obtain a bird's-eye view feature map.

[0100] Step 703: Perform lane line detection on the bird's-eye view to obtain the set of pixels and direction information corresponding to one or more lane lines contained in the bird's-eye view.

[0101] In some embodiments, a mobile device can input a bird's-eye view into a lane detection model, and the lane detection model can extract features from the bird's-eye view to obtain the set of pixels corresponding to each lane line in the bird's-eye view, as well as the direction information corresponding to each lane line.

[0102] The set of pixels corresponding to each lane line is determined based on multiple pixels in the bird's-eye view whose probability of belonging to a lane line is greater than a preset probability threshold.

[0103] The direction information corresponding to each lane line is determined based on the semantic information of the set of pixels corresponding to each lane line and the target arrangement order of multiple pixels in the set of pixels. The semantic information includes color and / or lane line type.

[0104] Furthermore, the road direction corresponding to each lane line is determined based on the semantic information of the pixel set corresponding to each lane line; the extension direction corresponding to each lane line is determined based on the target arrangement order of multiple pixels in the pixel set corresponding to each lane line.

[0105] As one implementation, the lane detection model includes an image decoder. The mobile device can input a bird's-eye view feature map into the image decoder, which performs dimensionality reduction decoding on the bird's-eye view feature map to transform the abstract features of the bird's-eye view into location and orientation information with clear physical meaning.

[0106] In some embodiments, the mobile device can input a bird's-eye view into a lane detection model, which then extracts features from the view to obtain the set of pixels corresponding to each lane line, as well as the pixel offset between each pixel in each set and its neighboring pixels. Based on the set of pixels corresponding to each lane line and the pixel offset between each pixel in each set and its neighboring pixels, the mobile device can determine the direction information corresponding to each lane line.

[0107] Mobile devices can determine the road direction corresponding to each lane line based on the set of pixels corresponding to each lane line contained in the bird's-eye view, determine the lane line offset corresponding to each lane line based on the pixel offset between each pixel in each pixel set and the neighboring pixels along the road direction, and determine the extension direction corresponding to each lane line based on the lane line offset corresponding to each lane line.

[0108] Figure 8 This is a flowchart illustrating lane line detection on a bird's-eye view, provided as an embodiment of this application. Figure 8 As shown, the steps involve inputting the bird's-eye view into the lane detection model, extracting features from the bird's-eye view using the lane detection model, obtaining the set of pixels corresponding to one or more lane lines contained in the bird's-eye view, as well as the direction information corresponding to each lane line. This may include the following steps: Step 802: Input the bird's-eye view into the lane line detection model, and extract features from the bird's-eye view through the lane line detection model to obtain the lane line heat map and pixel offset map.

[0109] In the lane heatmap, the pixel value of each pixel represents the probability that the pixel belongs to the lane line, and the pixel offset map represents the pixel offset between adjacent pixels in the lane heatmap.

[0110] The pixel offset map includes a first pixel offset map and a second pixel offset map. The first pixel offset map is used to characterize the pixel offset between adjacent pixels in the lane line heatmap on the first coordinate axis, and the second pixel offset map is used to characterize the pixel offset between adjacent pixels in the lane line heatmap on the second coordinate axis. The first coordinate axis is perpendicular to the second coordinate axis.

[0111] In some embodiments, the mobile device can input the bird's-eye view feature map into the image decoder, perform dimensionality reduction decoding on the bird's-eye view feature map through the image decoder, and output the lane line heat map, the first pixel offset map and the second pixel offset map through the three parallel output heads of the image decoder, respectively.

[0112] In some embodiments, a first pixel offset map can be used to characterize the pixel offset on a first coordinate axis between each pixel in the lane line heatmap and the next neighboring pixel along the road direction, and a second pixel offset map can be used to characterize the pixel offset on a second coordinate axis between each pixel in the lane line heatmap and the next neighboring pixel along the road direction.

[0113] Step 804: Based on the lane line heatmap, extract the lane lines from each lane line in the bird's-eye view to obtain the set of pixels corresponding to each lane line.

[0114] In some embodiments, the mobile device may extract multiple pixels corresponding to each lane line in sequence from one end to the other in the bird's-eye view to obtain a set of pixels corresponding to each lane line.

[0115] In some embodiments, a mobile device can extract pixels in a lane heatmap whose pixel values ​​are greater than or equal to a preset probability threshold to obtain a set of pixels corresponding to each lane line. Figure 9 This is a flowchart illustrating the process of determining the set of pixels corresponding to each lane line, as provided in an embodiment of this application. Figure 9 As shown, step 804 may include steps 901 to 905.

[0116] Step 901: Based on a preset probability threshold, the lane line heatmap is binarized to obtain the lane line foreground image.

[0117] Mobile devices can set pixels with values ​​greater than or equal to a preset probability threshold to 1 and pixels with values ​​less than the preset probability threshold to 0 in a lane heatmap, thus obtaining a binarized lane foreground image.

[0118] Step 903: Determine one or more connected regions contained in the lane line foreground image.

[0119] Mobile devices can determine one or more connected regions contained in the lane line foreground image based on the connectivity analysis algorithm corresponding to four-connected or eight-connected regions.

[0120] Step 905: Perform target processing operations on one or more connected regions to obtain a set of pixels corresponding to one or more lane lines.

[0121] The target processing operations include one or more of the following: skeleton refinement, pseudo-branch and branch removal, path sorting, and resampling.

[0122] Skeleton thinning can refer to shrinking or eroding each connected region in the lane line foreground image into a center line with a width of 1 pixel.

[0123] The process of removing false branches and forks can refer to traversing each branch of the centerline along the skeleton nodes and removing burr branches and abnormal forks with a length less than the length threshold, so as to obtain a centerline that only contains the trunk path.

[0124] Path sorting processing can refer to connecting the pixels on the center line sequentially according to their spatial proximity to form an ordered sequence of pixels from the start to the end of the lane line.

[0125] Resampling can refer to sampling an ordered sequence of pixels at preset intervals, either equidistantly or adaptively, to obtain a set of pixels that are evenly distributed and have consistent node spacing.

[0126] In one implementation, the mobile device can sequentially perform skeleton refinement, pseudo-branch and branch removal, path sorting and resampling on each connected region to obtain the set of pixels corresponding to each lane line.

[0127] Through the above implementation method, the set of pixels corresponding to lane lines can be determined according to the probability information in the lane line heat map, thereby obtaining the accurate position information of the continuous center line of each lane line. This helps to accurately determine the complete linear geometric structure of each lane line from the first image, ensuring that the generated lane line interface elements can accurately cover each lane line in the first image, avoiding visual misalignment or incomplete coverage between the lane line interface elements and the actual lane lines, which would affect the degree of fit between the lane line interface elements and the actual lane lines.

[0128] In some embodiments, the mobile device can obtain the set of pixels corresponding to each lane line by following the order of each lane line from one end to the other in the bird's-eye view and the pixels in the lane line heat map whose pixel values ​​are greater than or equal to a preset probability threshold.

[0129] Step 806: Determine the road direction corresponding to each lane line based on the set of pixels corresponding to each lane line.

[0130] In some embodiments, the mobile device may perform semantic segmentation on the set of pixels corresponding to each lane line, determine the color and / or lane line type corresponding to each lane line, and determine the road direction corresponding to each lane line based on the color and / or lane line type corresponding to each lane line.

[0131] Mobile devices can determine the color of each lane line based on the pixel values ​​of multiple pixels in each pixel set and their corresponding values ​​in the bird's-eye view. Furthermore, they can determine the lane line type based on the arrangement of these pixels in the bird's-eye view. For example, if multiple pixels in the pixel set are arranged continuously without gaps, the corresponding lane line type is determined to be a solid line; if they appear as regular, continuous segments, the corresponding lane line type is determined to be a dashed line.

[0132] Furthermore, the mobile device can determine the color and lane type of each lane line based on the set of pixels corresponding to each lane line in the bird's-eye view, determine the center lane line based on the color and lane type of each lane line, and determine the road direction corresponding to each lane line based on the lanes on both sides of the center lane line.

[0133] The center lane line refers to the lane line set in the middle of the road to separate lanes traveling in the same direction from lanes traveling in opposite directions.

[0134] Understandably, according to national standards and specifications for road traffic markings, the color and type of the center lane line used to separate opposing traffic flows are usually different from the color and type of the lane lines or lane edge lines that separate traffic flows in the same direction. For example, white lines are usually used to separate traffic flows in the same direction, while yellow lines (or double yellow lines) are usually used to separate opposing traffic flows. Therefore, by identifying the color and type of each lane line and determining the lane line that is different from the color and type of other lanes, the center lane line can be identified.

[0135] Furthermore, if the number of lane lines is 2, it can be determined that the current lane is a one-way lane or a two-way lane, and there is no center lane line; if the colors and lane line types of each lane line are the same, then there is no center lane line, and the lanes corresponding to each lane line are lanes in the same direction.

[0136] For example, in the case of three lane lines and two roads, if two lane lines are solid white lines and one lane line is dashed yellow lines, then the dashed yellow line can be identified as the center lane, one road is a lane in the same direction, and the other road is a lane in the opposite direction. If all three lane lines are either dashed yellow lines or dashed white lines, then both roads are lanes in the same direction, and there is no center lane.

[0137] In some embodiments, the mobile device can perform marker detection on the bird's-eye view, determine the directional arrows corresponding to one or more lanes, and determine the lane type corresponding to one or more lanes based on the directional arrows. The mobile device can determine the road direction corresponding to each lane line based on the lane type corresponding to each lane and the center lane line.

[0138] Step 808: Based on the set of pixels corresponding to each lane line, obtain the pixel offset corresponding to each set of pixels in the pixel offset map, and determine the lane line offset corresponding to each set of pixels based on the pixel offset corresponding to each set of pixels.

[0139] The lane line offset can be the total pixel offset between the first target pixel and the second target pixel in the set of pixels corresponding to each lane line.

[0140] In some embodiments, the mobile device may determine the pixel offset of each of the multiple target pixels in the target pixel set in the pixel offset map based on the target pixel set corresponding to the target lane line in the bird's-eye view, where the target lane line is any lane line.

[0141] Since the bird's-eye view, lane heatmap, and pixel offset map have the same dimensions, and the pixel value corresponding to each pixel in the pixel offset map is used to represent the pixel offset between adjacent pixels in the lane heatmap, the mobile device can read the pixel offset corresponding to each of the multiple target pixels in the target pixel set by indexing the same pixel coordinates in the pixel offset map based on the pixel coordinates corresponding to each pixel in the target pixel set in the bird's-eye view.

[0142] Mobile devices can calculate the sum of pixel offsets corresponding to multiple target pixels to obtain the cumulative offset between the first target pixel and the second target pixel in the target pixel set, which is used as the lane line offset corresponding to the target lane line. The first target pixel is the first target pixel in the target pixel set, and the second target pixel is the last target pixel in the target pixel set.

[0143] The mobile device can calculate the sum of the pixel offsets corresponding to the multiple target pixels in the initial arrangement order of the target pixel set, and obtain the cumulative offset as the lane line offset corresponding to the target lane line. The initial arrangement order refers to the order in which the mobile device extracts multiple pixels according to the order from one end of the lane line to the other.

[0144] In one implementation, the cumulative offset (i.e., lane line offset) includes a first cumulative offset corresponding to the first coordinate axis and a second cumulative offset corresponding to the second coordinate axis.

[0145] The mobile device can calculate the sum of the pixel offsets corresponding to the first coordinate axis of the multiple target pixels in the target pixel set according to the initial arrangement order of the multiple pixels in the target pixel set, to obtain the first accumulated offset, and calculate the sum of the pixel offsets corresponding to the second coordinate axis of the multiple target pixels in the target pixel set according to the sorting order of the multiple pixels in the target pixel set, to obtain the second accumulated offset.

[0146] Through the above implementation method, the accumulated offset is obtained by accumulating pixel by pixel along the target lane line. This can integrate the local changes of each pixel along the road direction on the target lane line, thus fully presenting the overall trend of the target lane line extending from the first target pixel along the road direction to the second target pixel. This helps to determine whether the extension direction of the target lane line under the initial arrangement order is the same as the corresponding road direction based on the accumulated offset, thereby providing an accurate and reliable direction derivation basis for the subsequent generation of lane line direction elements.

[0147] Step 810: Determine the extension direction of each lane line based on the lane line offset of each lane line contained in the bird's-eye view.

[0148] In some embodiments, the mobile device can determine the target arrangement order of the pixel set corresponding to each lane line based on the lane line offset corresponding to each lane line, and connect multiple pixels in each pixel set sequentially along the corresponding target arrangement order to form the direction as the extension direction corresponding to each lane line.

[0149] The target arrangement order is determined by the direction in which one pixel points to another pixel after the pixel moves through the lane line offset and falls into the preset range of another pixel.

[0150] The mobile device can tentatively move two pixels in the pixel set according to the lane offset corresponding to each lane line. If the moving pixel is within the preset range of the moving pixel, the target arrangement order is determined to be the order from the moving pixel to the moving pixel. If the moving pixel is within the preset range of the moving pixel, the target arrangement order is determined to be the order from the moving pixel to the moving pixel. The moving pixel refers to the pixel that is ranked earlier in the pixel set, and the moving pixel refers to the pixel that is ranked later in the pixel set.

[0151] In some embodiments, the preceding pixel may be a first target pixel and the following pixel may be a second target pixel, and the lane line offset includes a first accumulated offset and a second accumulated offset.

[0152] The mobile device can determine the target arrangement order of multiple target pixels in the target pixel set based on the first target pixel, the second target pixel, the first accumulated offset, and the second accumulated offset, and then connect the multiple target pixels in the target pixel set sequentially along the target arrangement order to form the direction of extension corresponding to the target lane line.

[0153] The target pixel arrangement order includes a first arrangement order or a second arrangement order. The first arrangement order is the order in which multiple target pixels in the target pixel set are arranged in a forward direction, and the second arrangement order is the order in which multiple target pixels in the target pixel set are arranged in a reverse direction. Multiple target pixels in the target pixel set are arranged sequentially along the first arrangement order, that is, the initial arrangement order is the first arrangement order. For example, if the target pixel set includes five pixels A, B, C, D, and E, the first arrangement order is ABCDE, and the second arrangement order is EDCBA.

[0154] In one implementation, when the initial arrangement order is the first arrangement order, the mobile device can tentatively move the first target pixel and the second target pixel according to the first accumulated offset and the second accumulated offset. If the first target pixel is within the preset range of the second target pixel after the movement, the target arrangement order is determined to be the first arrangement order. If the second target pixel is within the preset range of the first target pixel after the movement, the target arrangement order is determined to be the second arrangement order, which is the opposite of the first arrangement order. This ensures that the direction indicated by the lane line direction element constructed based on the target pixel is the same as the road direction of the lane line, thereby improving the accuracy of directional guidance through the lane line direction element.

[0155] Figure 10 A flowchart illustrating the determination of the target arrangement order provided in an embodiment of this application. For example... Figure 10 As shown, the step of determining the arrangement order of multiple target pixels in the target pixel set based on the first target pixel, the second target pixel, the first accumulated offset, and the second accumulated offset may include the following steps: Step 1002: Determine the first offset coordinates based on the first pixel coordinates, the first accumulated offset, and the second accumulated offset corresponding to the first target pixel.

[0156] With the first coordinate axis as the horizontal axis and the second coordinate axis as the vertical axis, the first pixel coordinate includes the first horizontal coordinate corresponding to the first coordinate axis and the first vertical coordinate corresponding to the second coordinate axis. The first offset coordinate includes the first offset horizontal coordinate corresponding to the first coordinate axis and the first offset vertical coordinate corresponding to the second coordinate axis.

[0157] The mobile device can obtain the first offset horizontal coordinate based on the sum of the first horizontal coordinate and the first accumulated offset, and obtain the first offset vertical coordinate based on the sum of the first vertical coordinate and the second accumulated offset.

[0158] Step 1004: Determine the second offset coordinates based on the second pixel coordinates corresponding to the second target pixel, the first accumulated offset, and the second accumulated offset.

[0159] The second pixel coordinates include a second x-coordinate corresponding to the first coordinate axis and a second y-coordinate corresponding to the second coordinate axis. The second offset coordinates include a second offset x-coordinate corresponding to the first coordinate axis and a second offset y-coordinate corresponding to the second coordinate axis.

[0160] The mobile device can obtain the second offset horizontal coordinate based on the sum of the second horizontal coordinate and the first accumulated offset, and obtain the second offset vertical coordinate based on the sum of the second vertical coordinate and the second accumulated offset.

[0161] Step 1006: Calculate the first distance difference between the first pixel coordinates and the second offset coordinates, and calculate the second distance difference between the second pixel coordinates and the first offset coordinates.

[0162] The mobile device can calculate the Euclidean distance between the first pixel coordinates and the second offset coordinates to obtain a first distance difference, and calculate the Euclidean distance between the second pixel coordinates and the first offset coordinates to obtain a second distance difference.

[0163] Step 1008: Determine whether the second distance difference is greater than the first distance difference. If not, proceed to step 1010; if yes, proceed to step 1012.

[0164] Understandably, the first pixel coordinate is the starting point, and the second pixel coordinate is the ending point. After introducing the cumulative offset, the first offset coordinate is essentially the position obtained by moving the starting point along the road direction by the cumulative offset, and therefore can be called the "pseudo-ending point". Similarly, the second offset coordinate can be called the "pseudo-starting point". Therefore, if the arrangement order is the same as the road direction, the starting point is moving closer to the ending point. The pseudo-ending point reached by the starting point after moving should be near or coincide with the ending point. The distance between the ending point and the pseudo-ending point should be less than the distance between the starting point and the pseudo-starting point. If the arrangement order is opposite to the road direction, the starting point is moving away from the ending point. The distance between the ending point and the pseudo-ending point should be greater than the distance between the starting point and the pseudo-starting point.

[0165] Step 1010: Determine the target arrangement order as the first arrangement order.

[0166] Step 1012: Determine the target arrangement order as the second arrangement order.

[0167] Figure 11AThis is a schematic diagram illustrating how to determine the order of targets, as provided in an embodiment of this application. Figure 11A As shown in (a), multiple target pixels of the target pixel set are arranged sequentially on the target lane line 1100 in a first arrangement order M. The mobile device can determine the first offset coordinate 1103 based on the first pixel coordinate 1101 corresponding to the first target pixel, and determine the second offset coordinate 1104 based on the second pixel coordinate 1102 corresponding to the second target pixel. It also calculates the first distance difference D1 between the first pixel coordinate 1101 and the second offset coordinate 1104, and the second distance difference D2 between the second pixel coordinate 1102 and the first offset coordinate 1103. If the second distance difference D2 is less than the first distance difference D1, it is determined that the first arrangement order M is the same as the actual road direction P of the target lane line 1100. Therefore, the target arrangement order can be determined as the first arrangement order M. Figure 11A As shown in (b), the direction formed by connecting multiple target pixels (solid hollow circles) in the target pixel set in sequence along the first arrangement order M is used as the extension direction corresponding to the target lane line 1100.

[0168] Figure 11B This is a schematic diagram illustrating another method for determining the arrangement order of targets, as provided in an embodiment of this application. Figure 11B As shown, multiple target pixels of the target pixel set are arranged on the target lane line 1100 in a second arrangement order N. The mobile device can determine the first offset coordinate 1107 based on the first pixel coordinate 1105 corresponding to the first target pixel, and determine the second offset coordinate 1108 based on the second pixel coordinate 1106 corresponding to the second target pixel. It can also calculate the first distance difference D1 between the first pixel coordinate 1105 and the second offset coordinate 1108, and the second distance difference D2 between the second pixel coordinate 1106 and the first offset coordinate 1107. If the second distance difference D2 is greater than the first distance difference D1, it is determined that the second arrangement order N is the same as the actual road direction P of the target lane line 1100. Therefore, the arrangement order of the target pixel set can be reversed from the second arrangement order N to the first arrangement order M so that the target arrangement order of the target pixel set corresponds to the road direction P of the target lane line 1100.

[0169] Through the above implementation method, it is possible to quickly and accurately determine whether the initial arrangement order of multiple target pixels in the target pixel set is consistent with the actual road direction of the target lane line, thereby accurately determining the direction indicated by the lane line direction element. While reducing computational complexity, it improves the real-time performance of lane line direction discrimination, helps to quickly display lane line interface elements, provides operators with the correct driving reference direction, reduces the risk of misjudgment of lane line direction, and enhances the operational safety of mobile devices.

[0170] In this embodiment, a bird's-eye view is input into the lane detection model, which simultaneously outputs a lane heatmap and a pixel offset map. The lane heatmap is used to locate the set of pixels corresponding to each lane line, while the pixel offset map provides the offset information of each pixel in the lane heatmap. Based on the set of pixels corresponding to each lane line, the road direction corresponding to each lane line is determined. Based on the set of pixels corresponding to each lane line and the pixel offset map, the overall lane line offset of each lane line is determined, thereby determining the extension direction of each lane line. This not only allows for the rapid and accurate determination of the road direction and extension direction of each lane line by fusing spatial location and pixel-level offset features, but also avoids the complex post-processing in traditional methods. It provides an accurate data foundation and alignment basis for the subsequent generation of lane line interface elements, thereby providing operators with the correct driving reference direction, reducing the risk of misjudging lane line direction, and enhancing the operational safety of mobile devices.

[0171] Step 705: Based on the conversion relationship between the bird's-eye view and each first image, map the set of pixels corresponding to each lane line to each first image to obtain the image position information and direction information of each lane line contained in each first image.

[0172] The transformation relationship between the bird's-eye view and each of the first images can be determined when the first images are stitched together to obtain the bird's-eye view. For example, a mobile device can calculate the corresponding homography matrix between the bird's-eye view and each of the first images to determine the transformation relationship between the bird's-eye view and each of the first images.

[0173] The pixel coordinates of each pixel in the pixel set are used to reflect the image position information of the lane line, and the arrangement order of each pixel in the pixel set is used to reflect the direction indicated by the lane line direction element, that is, the extension direction of the lane line.

[0174] In this embodiment, by stitching together the first images from multiple perspectives into a bird's-eye view and then uniformly detecting lane lines, the perspective distortion of the first images can be effectively eliminated, improving the detection accuracy and global consistency of lane lines in curves and distant areas, and avoiding the problems of lane line breakage and inconsistent direction caused by individual detection by multiple cameras. Furthermore, by using a unified transformation relationship to map the lane line detection results of the bird's-eye view back to each of the first images, the global consistency of the position and direction of lane lines in each of the first images is ensured, providing more accurate and reliable lane line data for the subsequent overlay display of lane line interface elements.

[0175] In some embodiments, the lane line interface elements include lane line identification elements and lane line direction elements. The mobile device can determine the lane line identification element based on the set of pixels corresponding to the lane line, and determine the lane line direction element based on the target arrangement order of multiple pixels in the set of pixels corresponding to the lane line.

[0176] Mobile devices can use lines that match the semantic information of the set of pixels corresponding to lane lines to sequentially connect multiple pixels in the set of pixels corresponding to lane lines to obtain lane line identifier elements.

[0177] If the lane lines in the image to be displayed are white and the lane line type is dashed, the mobile device can connect multiple pixels sequentially with white dashed lines to obtain the corresponding lane line identifier element; if the lane lines are yellow and the lane line type is double solid lines, the mobile device can connect multiple pixels sequentially with double yellow lines to obtain the corresponding lane line identifier element; if the lane line type is dashed-solid lines, the mobile device can connect multiple pixels corresponding to the dashed lines sequentially with dashed lines, and then connect multiple pixels corresponding to the solid lines sequentially with solid lines.

[0178] Or, such as Figure 3A As shown, a mobile device can use multiple pointer elements to connect multiple pixels in sequence, and each line segment in the dashed part is composed of multiple pointer elements connected together, while the blank space between two line segments does not have a pointer element.

[0179] Optionally, after determining one or more connected regions contained in the lane line foreground image, the mobile device can directly determine the set of connected regions corresponding to each lane line based on the adjacency and geometric continuity relationships between these regions, without performing target processing operations on each connected region. Subsequently, the mobile device can use the set of connected regions corresponding to each lane line as the corresponding lane line identifier element. For example, the mobile device can... Figure 6A The gray areas corresponding to the lane lines shown are all used as lane line identification elements.

[0180] In some embodiments, the mobile device can set a pointing element at the coordinates of each pixel in the image to be displayed, based on the direction that the pixel points to the next adjacent pixel in the set of pixels, which is the extension direction of the lane line, to obtain the lane line direction element.

[0181] As one implementation method, the spacing between multiple pointing elements can refer to the number of pixels between the pixels corresponding to the multiple pointing elements.

[0182] Mobile devices can select a subset of pixels from a pixel set based on a preset number of pixel intervals, using these pixels as anchor points for pointing elements. For example, a mobile device can select a pixel every 10 pixels and set the corresponding pointing element at that pixel.

[0183] Alternatively, the mobile device can determine the curvature of each pixel based on its individual pixel coordinates within the pixel set, and then determine the number of pixel intervals based on the curvatures of the individual pixels. For example, along a curve, the mobile device can select a pixel every 10 pixels, with the interval decreasing as the curve becomes more curved. Along a straight line, the mobile device can select a pixel every 20 pixels, thus completing the setting of the pointing element.

[0184] As one implementation method, the mobile device can determine the setting method of the pointing element corresponding to the lane line based on the lane line type.

[0185] If the lane line type is a single line, and the road directions on both sides of the lane line are the first road direction and the second road direction, then each directional element of the lane line includes a first sub-element in the same direction as the first road direction and a second sub-element in the same direction as the second road direction, wherein the first road direction and the second road direction are opposite.

[0186] By implementing the above methods, setting the pointing element based on the pixel point can make the lane line direction element closely fit the actual direction of the lane line. There is no need to predefine the arrow template or perform complex direction fitting. It can adapt to any lane line, improve the accuracy and visual coherence of the direction display, and help enhance the operator's intuitive understanding of the road direction.

[0187] Figure 12 This is a structural block diagram of a mobile device provided in an embodiment of this application. Figure 12 As shown, the mobile device 1200 may include a memory 1202 and a processor 1201. The memory 1202 stores a computer program. When the computer program is executed by the processor 1201, the mobile device 1200 implements the lane line display method as described in the above embodiments.

[0188] Processor 1201 may include one or more processing cores. Processor 1201 connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, processor 1201 may be implemented using at least one hardware form of digital signal processing, field-programmable gate array, or programmable logic array. Processor 1201 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1201 and may be implemented separately using a communication chip.

[0189] The memory 1202 may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created during the use of the computer device.

[0190] This application discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the lane line display method as described in the above embodiments.

[0191] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0192] This application discloses a lane line display device, which includes a computer program. When the computer program is executed by a processor, the processor implements the lane line display method as described in the above embodiments.

[0193] The above description is merely a specific example of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lane line display method characterized by, Applied to mobile devices, the method includes: Acquire the first image captured by multiple image acquisition devices on the mobile device; Lane line detection is performed on each of the first images to determine one or more lane lines contained in each of the first images; Based on one or more lane lines contained in the image to be displayed, determine the lane line interface element to be superimposed and displayed in the image to be displayed, wherein the image to be displayed is one or more of the first images. Based on each of the lane line interface elements, display the target image corresponding to each of the images to be displayed; The target image is an image on which lane line interface elements corresponding to each lane line are superimposed on an image to be displayed. The lane line interface elements are used to represent the position and / or type of the corresponding lane line, and to represent the road direction of the corresponding lane line.

2. The method of claim 1, wherein, The lane line interface elements include lane line identification elements and lane line direction elements. The lane line identification elements are used to represent the position and type of the corresponding lane line in the image to be displayed, and the lane line direction elements are used to represent the road direction of the lane line.

3. The method of claim 2, wherein, The color of the lane line identifier element is the same as the color of the corresponding lane line, and the lane line type of the lane line identifier element matches the lane line type of the corresponding lane line. or, The color of the lane line marking element includes a first color and a second color. The first color is used to represent the first lane line type of the lane line corresponding to the lane line marking element, and the second color is used to represent the second lane line type of the lane line corresponding to the lane line marking element.

4. The method of claim 2, wherein, The lane line direction element includes multiple pointing elements, which are arranged sequentially and at intervals along the road direction of the corresponding lane line on the corresponding image to be displayed.

5. The method of claim 4, wherein, The spacing between the plurality of pointing elements is determined based on the curvature of the lane line corresponding to the lane line identifier element.

6. The method of claim 1, wherein, The display of the target image corresponding to each of the images to be displayed includes: At the positions corresponding to each lane line contained in each of the images to be displayed, the lane line interface elements corresponding to each lane line are overlaid and displayed.

7. The method according to any one of claims 1 to 6, characterized in that, The step of performing lane line detection on each of the first images to determine one or more lane lines contained in each of the first images includes: Lane line detection is performed on each of the first images to determine the image position information and direction information corresponding to one or more lane lines contained in each of the first images. The step of determining the lane line interface element to be superimposed and displayed based on one or more lane lines contained in the image to be displayed includes: Based on the image position and direction information corresponding to each lane line in the image to be displayed, a lane line interface element is generated to be superimposed and displayed on each lane line in the image to be displayed.

8. The method of claim 7, wherein, The step of performing lane line detection on each of the first images to determine the image position and direction information corresponding to one or more lane lines contained in each of the first images includes: The first images are stitched together to obtain a bird's-eye view; Lane line detection is performed on the bird's-eye view to obtain the set of pixels and direction information corresponding to one or more lane lines contained in the bird's-eye view; Based on the conversion relationship between the bird's-eye view and each of the first images, the set of pixels corresponding to each lane line is mapped to each of the first images to obtain the image position information and direction information corresponding to each lane line contained in each of the first images.

9. The method of claim 8, wherein, The process of detecting lane lines on the bird's-eye view to obtain the set of pixels and direction information corresponding to one or more lane lines in the bird's-eye view includes: The bird's-eye view is input into the lane detection model, and the model extracts features from the bird's-eye view to obtain the set of pixels corresponding to one or more lane lines in the bird's-eye view, as well as the direction information corresponding to each lane line.

10. The method of claim 9, wherein, The direction information includes the road direction corresponding to the lane line and the extension direction corresponding to the lane line; The step involves inputting the bird's-eye view into a lane detection model, and then using the lane detection model to extract features from the bird's-eye view to obtain a set of pixels corresponding to one or more lane lines in the bird's-eye view, as well as the direction information corresponding to each lane line, including: The bird's-eye view is input into the lane line detection model, and the lane line detection model performs feature extraction on the bird's-eye view to obtain a lane line heatmap and a pixel offset map. The pixel value of each pixel in the lane line heatmap is used to represent the probability that the pixel belongs to the lane line, and the pixel offset map is used to represent the pixel offset between adjacent pixels in the lane line heatmap. Based on the lane heatmap, lane lines are extracted from each lane line in the bird's-eye view to obtain the set of pixels corresponding to each lane line. Based on the set of pixels corresponding to each lane line, the road direction corresponding to each lane line is determined; Based on the set of pixels corresponding to each lane line, obtain the pixel offset corresponding to each set of pixels in the pixel offset map, and determine the lane line offset corresponding to each set of pixels based on the pixel offset corresponding to each set of pixels. Based on the lane offset of each lane line contained in the bird's-eye view, the extension direction of each lane line is determined.

11. The method of claim 10, wherein, The step involves extracting lane lines from the bird's-eye view based on the lane line heatmap, obtaining a set of pixels corresponding to each lane line, including: Based on a preset probability threshold, the lane heatmap is binarized to obtain a lane foreground image. Identify one or more connected regions contained in the lane line foreground image; Target processing operations are performed on the one or more connected regions to obtain a set of pixels corresponding to one or more lane lines. The target processing operations include one or more of the following: skeleton thinning, pseudo-branch and branch removal, path sorting, and resampling.

12. The method of claim 10, wherein, The step of obtaining the pixel offset corresponding to each pixel set in the pixel offset map based on the pixel set corresponding to each lane line, and determining the lane line offset corresponding to each pixel set based on the pixel offset corresponding to each pixel set, includes: Based on the set of target pixels corresponding to the target lane line in the bird's-eye view, the pixel offsets corresponding to the multiple target pixels included in the target pixel set are determined in the pixel offset map, and the target lane line is any of the lane lines. The sum of the pixel offsets corresponding to the plurality of target pixels is calculated to obtain the cumulative offset between the first target pixel and the second target pixel in the target pixel set, which is used as the lane line offset corresponding to the target lane line. The first target pixel is the first target pixel in the target pixel set, and the second target pixel is the last target pixel in the target pixel set.

13. The method of claim 12, wherein, The pixel offset map includes a first pixel offset map and a second pixel offset map. The first pixel offset map is used to characterize the pixel offset between adjacent pixels in the lane line heat map on the first coordinate axis. The second pixel offset map is used to characterize the pixel offset between adjacent pixels in the lane line heat map on the second coordinate axis. The first coordinate axis is perpendicular to the second coordinate axis. The accumulated offset includes a first accumulated offset corresponding to the first coordinate axis and a second accumulated offset corresponding to the second coordinate axis; The step of determining the extension direction of each lane line based on the lane line offset of each lane line included in the bird's-eye view includes: Based on the first target pixel, the second target pixel, the first accumulated offset, and the second accumulated offset, the target arrangement order of multiple target pixels in the target pixel set is determined, and the direction formed by connecting multiple target pixels in the target pixel set sequentially along the target arrangement order is used as the extension direction corresponding to the target lane line.

14. The method of claim 13, wherein, The target arrangement order includes a first arrangement order or a second arrangement order. The first arrangement order is the order in which multiple target pixels in the target pixel set are arranged in a forward direction, and the second arrangement order is the order in which multiple target pixels in the target pixel set are arranged in a reverse direction. The multiple target pixels in the target pixel set are arranged sequentially along the first arrangement order. Determining the target arrangement order of multiple target pixels in the target pixel set based on the first target pixel, the second target pixel, the first accumulated offset, and the second accumulated offset includes: The first offset coordinate is determined based on the first pixel coordinates corresponding to the first target pixel, the first accumulated offset, and the second accumulated offset. The second offset coordinate is determined based on the second pixel coordinates corresponding to the second target pixel, the first accumulated offset, and the second accumulated offset. Calculate the first distance difference between the first pixel coordinates and the second offset coordinates, and calculate the second distance difference between the second pixel coordinates and the first offset coordinates; If the second distance difference is less than or equal to the first distance difference, then the target arrangement order is determined to be the first arrangement order; If the second distance difference is greater than the first distance difference, then the target arrangement order is determined to be the second arrangement order.

15. A mobile device, comprising: The method includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 14.

16. A lane line display device comprising a computer program, characterized by When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 14.