A method, storage medium, and electronic device for lane line recognition of a rubber tyred gantry crane

Abstract

The present invention provides a method, storage medium and electronic device for lane line recognition of a tire crane. The method includes the following steps: training a lane line recognition model to obtain a trained lane line recognition model; collecting lane data images, inputting the lane data images into the trained lane line recognition model to identify lane line positions; and determining the driving offset parameters of the current tire crane based on comparison of the lane line positions with predetermined baselines. When training the model, an IOU value output prediction for the sample data is added to the regression output of the model. If the IOU value of the sample data output prediction does not exist, the sample data is eliminated when the lane line recognition model is trained next time. By setting the IOU value to filter the training sample data, the recognition accuracy of the model can be effectively improved, thereby improving the accuracy of lane line recognition in actual application scenarios, and facilitating adjustment of the driving parameters of the tire crane.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method, storage medium and electronic device for lane line recognition of a tire crane. Background Art

[0002] As intelligent systems are applied to vehicle driving, more and more vehicles are equipped with intelligent systems that enable autonomous driving or assisted driving. To achieve these functions, the intelligent systems on vehicles typically need to identify lane lines from images of the road surrounding the vehicle to ensure that the vehicle can drive within the required lane lines.

[0003] Tire-tire cranes, including rubber-tired gantry cranes (RTGs), are prone to deviating from their intended path due to factors such as weight imbalance, unsuitable dock roads, and motor output delays. In these situations, the driver must manually correct the deviation or use visual lane recognition and geolocation to automatically control the vehicle's deviation.

[0004] The LaneATT model is one of the models of the visual recognition algorithm. During the model training process, the model calculates a confidence probability value for each sample data and uses the confidence probability value as the basis for whether the sample data participates in the next training. However, after being activated by the activation function, the confidence probability value will be concentrated at the two ends of 0 and 1, making it difficult for the probability value in the middle segment to appear, causing the confidence to deviate from the actual situation and affecting the accuracy of the model prediction. Summary of the Invention

[0005] Therefore, it is necessary to provide a technical solution for lane line recognition to solve the technical problem of poor prediction accuracy in the existing lane line recognition process.

[0006] In a first aspect, the present application provides a method for lane line recognition of a tire crane, the method comprising the following steps:

[0007] S1: Train the lane recognition model to obtain a trained lane recognition model; when training the model, add an output prediction of the IOU value of the sample data to the regression output of the model; if the IOU value of the sample data output prediction does not exist, then the sample data is discarded when the lane recognition model is trained next time;

[0008] The calculation formula of the IOU value is as follows:

[0009]

[0010] Among them, D(Li , l j ) represents the distance between the predicted line and the labeled line calculated based on the sample data, D th Indicates the preset threshold, None indicates that the IOU value does not exist;

[0011] S2: Collect lane data images and input the lane data images into the trained lane line recognition model to identify the lane line positions;

[0012] S3: Determine a current driving offset parameter of the tire crane based on a comparison between the lane line position and a predetermined reference line.

[0013] As an optional embodiment, the current tire crane is provided with an image acquisition unit in the traveling direction, and the image acquisition unit is used to acquire the lane data image.

[0014] As an optional embodiment, the method includes:

[0015] The lane data image is cropped according to the shooting direction of the image acquisition unit to obtain a cropped image, and the cropped image is input into the trained lane line recognition model; the cropped image contains the lane line.

[0016] As an optional embodiment, the offset parameters include a deflection angle and an offset distance;

[0017] The method further comprises:

[0018] S4: adjusting the current traveling direction of the tire crane based on the deflection angle and the offset distance until the lane position identified based on the re-collected lane data image meets the predetermined reference line.

[0019] As an optional embodiment, the lane line is a pair of straight lines arranged side by side in parallel, including a first lane line and a second lane line; the predetermined reference line is a reference center line;

[0020] Determining the current tire crane travel offset parameter based on the comparison between the lane line position and a predetermined reference line includes:

[0021] A corresponding middle line is fitted based on the identified first lane straight line and second lane straight line, the middle line is compared with a reference center line, an offset parameter of the middle line compared to the reference center line is calculated, and the offset parameter is determined as the current travel offset parameter of the tire crane.

[0022] As an optional embodiment, fitting a corresponding middle line based on the identified first lane straight line and second lane straight line includes:

[0023] The coordinates of the midline points are calculated according to the following formula:

[0024]

[0025] Among them, (x c ,y c ) are the coordinates of the midline point, (x1, y1) are the coordinates of the point on the first lane line, (x2, y2) are the coordinates of the point on the second lane line, and n represents that one corner of the lane data image is placed at the origin of the coordinate system and is divided into n parts along the Y-axis direction of the coordinate system;

[0026] The least square method is used to fit all the middle line coordinates into a straight line to obtain the middle line.

[0027] As an optional embodiment,

[0028]

[0029] Among them, X p Indicates the distance between the point on the prediction line and the horizontal axis, x p Indicates the standard line and X P The distance between the corresponding point and the horizontal axis, s c =max(s L i ,sl j ) indicates the start and end index of the point coordinates of the prediction line, e c =min(inf,el j)=el j represents the index start and end index of the point coordinates of the annotation line.

[0030] As an optional embodiment, the method includes:

[0031] When the clarity of the lane line in the lane data image cannot meet the preset clarity requirement through the self-attention module, the lane line position is predicted based on the feature points around the lane line in the lane data image, and identification information is added to the predicted lane line.

[0032] In a second aspect, the present invention further provides a storage medium storing a computer program, which implements the method steps of the first aspect of the present invention when executed by a processor.

[0033] In a third aspect, the present invention further provides an electronic device, comprising a processor and a storage medium, wherein the storage medium is the storage medium of the second aspect;

[0034] The processor is configured to execute a computer program stored in a storage medium to implement the method steps of the first aspect.

[0035] Different from the prior art, the present invention provides a method, storage medium and electronic device for lane line recognition of a tire crane, the method comprising the following steps: training a lane line recognition model to obtain a trained lane line recognition model; collecting lane data images, inputting the lane data images into the trained lane line recognition model to identify the lane line positions; and determining the driving offset parameters of the current tire crane based on the comparison of the lane line positions with a predetermined baseline. When training the model, an IOU value output prediction for the sample data is added to the regression output of the model. If the IOU value of the sample data output prediction does not exist, the sample data is discarded when the lane line recognition model is trained next time. By setting the IOU value to filter the training sample data, the recognition accuracy of the model can be effectively improved, thereby improving the accuracy of lane line recognition in actual application scenarios, and facilitating the adjustment of the driving parameters of the tire crane. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for lane line recognition of a tire crane according to a first exemplary embodiment of the present invention;

[0037] Figure 2 This is a flow chart of a method for lane line recognition of a tire crane according to a second exemplary embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of a lane data image involved in an exemplary embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of a cropped image according to an exemplary embodiment of the present invention;

[0040] Figure 5 Schematic diagram of a lane data image after segmentation according to an exemplary embodiment of the present invention;

[0041] Figure 6 A schematic diagram of a fitted midline prediction according to an exemplary embodiment of the present invention;

[0042] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention;

[0043] Reference numerals:

[0044] 20. First lane straight line;

[0045] 30. Second lane straight line;

[0046] 40. Middle line;

[0047] 50. Reference center line;

[0048] 201, deflection angle;

[0049] 202, offset distance;

[0050] 10. Electronic equipment;

[0051] 101. Processor;

[0052] 102. Storage medium. DETAILED DESCRIPTION

[0053] In order to explain in detail the possible application scenarios, technical principles, specific solutions that can be implemented, and the purpose and effects of this application, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this application and are therefore only examples and are not intended to limit the scope of protection of this application.

[0054] References to "embodiments" herein mean that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the word "embodiment" in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or relevance to other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the various technical features mentioned in the embodiments can be combined in any manner to form a corresponding implementable technical solution.

[0055] Unless otherwise defined, the technical terms used herein have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms herein is only for describing specific embodiments and is not intended to limit this application.

[0056] In the description of this application, the term "and / or" is used to describe a logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and both A and B exist. In addition, the character " / " in this document generally indicates that the objects before and after are in a logical "or" relationship.

[0057] In this application, terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, priority or sequence relationship between these entities or operations.

[0058] Without further limitations, in this application, the words "include", "comprise", "have" or other similar expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product that includes the elements, so that the process, method or product that includes a series of elements may include not only those limited elements, but also other elements that are not explicitly listed, or also include elements that are inherent to such process, method or product.

[0059] Consistent with the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceed" are understood to exclude the number itself; expressions such as "above," "below," and "within" are understood to include the number itself. Furthermore, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this manner, such as "multiple groups," "multiple times," etc., unless otherwise specifically defined.

[0060] like Figure 1 As shown, in a first aspect, the present invention provides a method for lane line recognition of a tire crane, comprising the following steps:

[0061] S1: Train the lane recognition model to obtain a trained lane recognition model; when training the model, add an output prediction of the IOU value of the sample data to the regression output of the model; if the IOU value of the sample data output prediction does not exist, then the sample data is discarded when the lane recognition model is trained next time;

[0062] The calculation formula of the IOU value is as follows:

[0063]

[0064] Among them, D(L i , l j ) represents the distance between the predicted line and the labeled line calculated based on the sample data, D th Indicates the preset threshold, None indicates that the IOU value does not exist;

[0065] S2: Collect lane data images and input the lane data images into the trained lane line recognition model to identify the lane line positions;

[0066] S3: Determine a current driving offset parameter of the tire crane based on a comparison between the lane line position and a predetermined reference line.

[0067] In this embodiment, D(L i , l j) is larger, indicating that the distance between the prediction line and the annotation line calculated based on the sample data is farther. This part of the sample data will participate in the gradient training of the subsequent model, which will affect the recognition accuracy of the model. This application sets a preset threshold D th By comparing the IOU values of each sample data obtained after training, sample data with a large distance between the prediction line and the annotation line are eliminated, which can effectively improve the training accuracy of the model.

[0068] In this embodiment, a lane data image refers to a data image of the road ahead or behind the crane, captured while the crane is in motion. Typically, lane markings are set on the road the crane is traveling on, and the crane must operate along these lane markings to comply with operational specifications. The current driving deviation parameter of the crane refers to the degree to which the wheels of the crane deviate from the lane markings during its current driving process.

[0069] In certain embodiments, the current tire crane is equipped with an image acquisition unit in the direction of travel, and the image acquisition unit is used to capture the lane data image. The image acquisition unit is a device with image data acquisition capabilities, and can be a camera or a device with a photographic function. Preferably, the image acquisition unit is mounted in a fixed position in front of the tire crane. This ensures that the image acquisition unit always captures the road data image (i.e., lane data image) in the direction of travel of the tire crane, facilitating subsequent analysis of the road data image to determine the current tire crane travel offset parameter.

[0070] In certain embodiments, the method includes: cropping the lane data image according to the shooting direction of the image acquisition unit to obtain a cropped image, and inputting the cropped image into a trained lane line recognition model; the cropped image includes the lane line.

[0071] like Figure 3 FIG. 1 is a schematic diagram of a lane data image according to an exemplary embodiment of the present invention. Figure 3It is not difficult to see that the lane data image captured by the image acquisition unit contains not only the lane line information of the tire crane, but also a large amount of irrelevant background information (such as the fence on the side of the road, the manhole cover on the floor, etc.). If the lane data image is directly input into the recognition model, the computational complexity of the recognition model will increase, affecting the computational efficiency and accuracy. Therefore, the present application crops the lane data image according to the shooting direction of the image acquisition unit to obtain a cropped image, and inputs the cropped image into the trained lane line recognition model, which can effectively reduce the computational complexity of the recognition model and improve the recognition accuracy. Preferably, the approximate position of the lane line in the original lane data image can be determined by grayscale recognition, and then the lane line position can be widened outward by M unit size (the M value can be set in advance), and then the original lane data image is cropped to obtain a cropped image.

[0072] Crop the image as Figure 4 shown.

[0073] In some embodiments, the offset parameters include a deflection angle and an offset distance. Figure 2 As shown, the method further includes the following steps after step S3: S4: adjusting the current travel direction of the tire crane based on the deflection angle and offset distance until the lane position identified based on the re-collected lane data image meets the predetermined baseline. Specifically, when a deviation in the travel direction of the tire crane is detected, after calculating the deflection angle and offset distance, the deflection angle and offset distance can be sent to the control system of the tire crane. The tire crane control system adjusts the travel direction of the tire crane through the drive unit to achieve automatic deviation correction of the tire crane. In order to achieve the predetermined purpose of the deviation correction, during the process of adjusting the travel direction of the tire crane, the camera located in front of the tire crane can also collect lane data images in real time, and the real-time collected lane data images are re-input into the recognition model for judgment until the lane position identified based on the re-collected lane data images meets the predetermined baseline, that is, to ensure that the tire crane can continue to travel along the lane line after adjustment.

[0074] In certain embodiments, as Figure 6 As shown, the lane lines are two parallel straight lines, including a first lane line 20 and a second lane line 30; the predetermined reference line is a reference center line 50; based on the comparison between the lane line position and the predetermined reference line, the current tire crane driving offset parameter is determined including:

[0075] Based on the identified first lane straight line 20 and second lane straight line 30, a corresponding middle line 40 is fitted, the middle line 40 is compared with the reference center line 50, and the offset parameter of the middle line 40 compared to the reference center line 50 is calculated. The offset parameter is determined as the current travel offset parameter of the tire crane. The offset parameter includes an offset distance 202 and a deflection angle 201, for example Figure 6 Angle A represents the angle between the fitted centerline 40 and the reference centerline 50, and offset distance 202 represents the relative distance between the fitted centerline 40 and the reference centerline 50. This means that the fitted centerline 40 is shifted a certain distance along the deflection angle so that the centerline 40 and the reference centerline 50 coincide. Because the offset distance varies between different coordinate points on the centerline 40 and the reference centerline 50, the offset distance can be calculated by selecting a point on the centerline 40 and the reference centerline 50 with the same vertical coordinate (the vertical axis of the coordinate system is located along the reference centerline), thereby simplifying the calculation.

[0076] In certain embodiments, as Figure 5 As shown, fitting the corresponding middle line based on the identified first lane straight line and second lane straight line includes:

[0077] The coordinates of the midline points are calculated according to the following formula:

[0078]

[0079] Among them, (x c ,y c ) are the coordinates of the midline point, (x1, y1) are the coordinates of the point on the first lane line, (x2, y2) are the coordinates of the point on the second lane line, and n represents that one corner of the lane data image is placed at the origin of the coordinate system and then divided (preferably equally) into n parts along the Y-axis direction of the coordinate system;

[0080] The least square method is used to fit all the middle line coordinates into a straight line to obtain the middle line.

[0081] In short, after dividing the image into n parts along the Y axis, the intersection of the lane lines on the image and the dividing lines can be used as the lane line feature points. Each lane line can then be represented by the ordinates of several feature points (i.e., the distance from the horizontal axis). By identifying and calculating these feature points, the corresponding lane line position on the image can be obtained. Since the coordinates of the midline points of the feature points on two discrete lane lines may not be on the same straight line when calculating, they can be fitted to a straight line using the least squares method to obtain the said midline, which facilitates the subsequent calculation of the deflection angle and offset distance.

[0082] In this embodiment,

[0083]

[0084] Among them, X p Indicates the distance between the point on the prediction line and the horizontal axis, x p Indicates the standard line and X P The distance between the corresponding point and the horizontal axis, s c =max(s L i ,sl j ) indicates the start and end index of the point coordinates of the prediction line, e c =min(inf,el j)=el j represents the index start and end index of the point coordinates of the annotation line.

[0085] The standard line refers to the standard value of the model. That is, the closer the predicted line is to the standard line, the better the effect of this model training is. For the distance comparison between the standard line and the predicted line, after each sample image is passed through Figure 5 After segmentation as shown, the distance between the two can be converted to the distance between the feature points on the standard line and the feature points on the predicted line relative to the horizontal axis. For example, if the image is segmented into 72 points, the distance between the predicted line and the standard line obtained from the sample image data can be calculated from the index 0 to the 71st feature point.

[0086] In certain embodiments, the method includes: when the clarity of the lane line in the lane data image fails to meet the preset clarity requirement, predicting the lane line position based on feature points around the lane line in the lane data image by a self-attention module, and adding identification information to the predicted lane line. Preferably, in this embodiment, the lane line is a straight line. Usually, the lane line is marked with different colors on the image, but due to long-term wear between the lane line and the tire of the tire crane, the lane line on the captured image becomes blurred. For example, only a part of the lane line is left with a discontinuous line segment, but the color of this part of the line segment is still different from the other parts of the image. Therefore, the RGB component values on the image can be extracted by identifying and extracting them. After determining the position of the discontinuous point of the lane line, the lane line is restored by reconnecting these points, and identification information is added to the predicted lane line, such as highlighting it with an identification of the same color as the lane line, so that the clarity of the predicted lane line can meet the preset clarity requirement again.

[0087] In a second aspect, the present invention further provides a storage medium storing a computer program, which implements the method steps of the first aspect of the present invention when executed by a processor.

[0088] like Figure 7As shown, in the third aspect, the present invention also provides an electronic device 10, including a processor 101 and a storage medium 102, wherein the storage medium 102 is the storage medium as in the second aspect; the processor 101 is used to execute the computer program stored in the storage medium 102 to implement the method steps as in the first aspect.

[0089] In this embodiment, the electronic device is a computer device, including but not limited to: oral scanners, personal computers, servers, general-purpose computers, special-purpose computers, network devices, embedded devices, programmable devices, smart mobile terminals, smart home devices, wearable smart devices, in-vehicle smart devices, etc. Storage media include but are not limited to: RAM, ROM, magnetic disks, tapes, optical disks, flash memory, USB flash drives, mobile hard drives, memory cards, memory sticks, network server storage, network cloud storage, etc. Processors include but are not limited to CPUs (central processing units), GPUs (graphics processing units), MCUs (microprocessors), etc.

[0090] Those skilled in the art will appreciate that the above embodiments can be provided as methods, apparatuses, or computer program products. These embodiments can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. All or part of the steps in the methods involved in the above embodiments can be completed by a program instructing the relevant hardware. The program can be stored in a computer-readable storage medium for executing all or part of the steps in the methods of the above embodiments.

[0091] The above embodiments are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of the processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a computer device to generate a machine, so that the instructions executed by the processor of the computer device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions can also be stored in a computer device readable memory that can guide a computer device to work in a specific manner, so that the instructions stored in the computer device readable memory produce a product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer device so that a series of operating steps are executed on the computer device to produce a computer-implemented process, whereby the instructions executed on the computer device provide for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] Although the above embodiments have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concepts. Therefore, the above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present invention.

Claims

1. A method for lane line recognition of a tire crane, characterized in that: The method comprises the following steps: S1: Train the lane recognition model to obtain a trained lane recognition model; when training the model, add an output prediction of the IOU value of the sample data to the regression output of the model; if the IOU value of the sample data output prediction does not exist, then the sample data is discarded when the lane recognition model is trained next time; The calculation formula of the IOU value is as follows: Among them, D(L i , l j ) represents the distance between the predicted line and the labeled line calculated based on the sample data, D th Indicates the preset threshold, None indicates that the IOU value does not exist; S2: Collect lane data images and input the lane data images into the trained lane line recognition model to identify the lane line positions; S3: Determine a current driving offset parameter of the tire crane based on a comparison between the lane line position and a predetermined reference line.

2. The method for lane line recognition of a tire crane according to claim 1, characterized in that: The current tire crane is provided with an image acquisition unit in the traveling direction, and the image acquisition unit is used to acquire the lane data image.

3. The method for lane line recognition of a tire crane according to claim 2, characterized in that: The method comprises: The lane data image is cropped according to the shooting direction of the image acquisition unit to obtain a cropped image, and the cropped image is input into the trained lane line recognition model; the cropped image contains the lane line.

4. The method for lane line recognition of a tire crane according to claim 1 or 2, characterized in that: The offset parameters include a deflection angle and an offset distance; The method further comprises: S4: adjusting the current traveling direction of the tire crane based on the deflection angle and the offset distance until the lane position identified based on the re-collected lane data image meets the predetermined reference line.

5. The method for lane line recognition of a tire crane according to claim 1, wherein: The lane lines are two straight lines arranged side by side in parallel, including a first lane line and a second lane line; the predetermined reference line is a reference center line; Determining the current tire crane travel offset parameter based on the comparison between the lane line position and a predetermined reference line includes: A corresponding middle line is fitted based on the identified first lane straight line and second lane straight line, the middle line is compared with a reference center line, an offset parameter of the middle line compared to the reference center line is calculated, and the offset parameter is determined as the current travel offset parameter of the tire crane.

6. The method for lane line recognition of a tire crane according to claim 5, characterized in that: Fitting the corresponding middle line based on the identified first lane straight line and second lane straight line includes: The coordinates of the midline points are calculated according to the following formula: Among them, (x c ,y c ) are the coordinates of the midline point, (x1, y1) are the coordinates of the point on the first lane line, (x2, y2) are the coordinates of the point on the second lane line, and n represents that one corner of the lane data image is placed at the origin of the coordinate system and is divided into n parts along the Y-axis direction of the coordinate system; The least square method is used to fit all the middle line coordinates into a straight line to obtain the middle line.

7. The method for lane line recognition of a tire crane according to claim 1, characterized in that: Among them, X p Indicates the distance between the point on the prediction line and the horizontal axis, x p Indicates the standard line and X P The distance between the corresponding point and the horizontal axis, s c =max(s L i ,sl j ) indicates the start and end index of the point coordinates of the prediction line, e c =min(inf,el j)=elj represents the index start and end index of the point coordinates of the annotation line.

8. The method for lane line recognition of a tire crane according to claim 1, wherein: The method comprises: When the clarity of the lane line in the lane data image cannot meet the preset clarity requirement through the self-attention module, the lane line position is predicted based on the feature points around the lane line in the lane data image, and identification information is added to the predicted lane line.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 8 are implemented.

10. An electronic device, characterized in that: comprising a processor and a storage medium, wherein the storage medium is the storage medium according to claim 9; The processor is configured to execute the computer program stored in the storage medium to implement the method steps according to any one of claims 1 to 8.

Images