Motor home hook identification and positioning method and system based on binocular camera

Abstract

The invention discloses a motor home hook recognition and positioning method and system based on a binocular camera. The method comprises the steps that a motor home hook picture data set is collected and made; preprocessing the touring car hook picture data set; performing model training according to the preprocessed touring car hook picture data set to form a first recognition model for preliminarily recognizing a first range in a touring car hook area and a second recognition model for performing key point feature recognition in the first range; obtaining a to-be-detected image and a binocular image, processing the to-be-detected image through the first recognition model and the second recognition model, and obtaining key point feature data in the to-be-detected image; pixel position extraction is carried out on the key point feature data; according to the pixel position information of the key point feature data and the binocular image, the actual position information of the motor home hook is obtained, and the motor home hook is identified by adopting a camera, so that the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method and system for identifying and locating a motorhome hook based on a binocular camera. Background Art

[0002] With the rise of RV tourism, the demand for automated docking between trailer RVs and family cars is becoming increasingly strong, which greatly improves the convenience and comfort of users. In the current technical field, commonly used sensor types are divided into the LiDAR series.

[0003] At present, due to the high price of high-beam radars, low-beam radars have sparse point clouds in the process of identifying RV hooks, making it difficult to identify and locate RV hooks. Summary of the invention

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention discloses a method for identifying and locating a RV hook based on a binocular camera, comprising: Collect and create a dataset of RV hook images; Preprocess the RV hook image dataset; Model training is performed based on the preprocessed RV hook image data set to form a first recognition model for preliminarily identifying a first range within the RV hook area, and a second recognition model for identifying key point features within the first range; Acquire an image to be detected and a binocular image, and process the image to be detected by using a first recognition model and a second recognition model to obtain key point feature data in the image to be detected; Extract pixel positions of key point feature data; The actual position information of the RV hook is obtained based on the pixel position information of the key point feature data and the binocular image.

[0005] Furthermore, the key point of the RV hook is the ball head of the hook.

[0006] Furthermore, the preprocessing of the RV hook image dataset includes: Perform sample selection on the RV hook image dataset; Annotate the images according to the requirements of the first recognition model and the second recognition model respectively; Perform data cleaning and data enhancement on the image dataset.

[0007] Furthermore, the establishment of the first recognition model includes: Set the data organization format to coco data format; The transformer-based DETR target detection algorithm is used as the model training algorithm; Among them, the Resnet used in the backbone of the DETR network structure is replaced by MobileNetV3.

[0008] Furthermore, the second recognition model uses a model training based on key point features based on YOLOV8.

[0009] Furthermore, after the first recognition model and the second recognition model are established, a quantization acceleration operation is performed on the first recognition model and the second recognition model, including: INT8 is used for model quantization.

[0010] Furthermore, the obtaining of the actual position information of the RV hook according to the pixel position information of the key point feature data and the binocular image includes: Calibrate the binocular camera; Correct the original binocular image according to the calibration results; Extract disparity map; According to the extracted disparity map, a depth map is calculated; Combined with the pixel position information of the key point feature data, the actual position information of the RV hook is extracted from the depth map.

[0011] Furthermore, extracting the disparity map includes: Match the corresponding points between the left and right images of the binocular image based on SGBM: ; in, It represents the calculation of the neighborhood gradient around a point in the pixel coordinate system; In calculating the cost of pixel matching, the neighborhood summation method is used, that is, the matching cost of each pixel will include information about the surrounding local area: ; in, Represents a point p in the field The sum of all the cost calculations of the surrounding neighborhood is Represents the window size.

[0012] Further, the calculating the depth map according to the extracted disparity map includes: ; Among them, z represents the specific depth information represented by the pixel, f represents the focal length of the camera, b represents the baseline length of the camera, and d represents the parallax, that is, the horizontal distance difference between the object on the left and right camera imaging planes.

[0013] On the other hand, the present invention also provides a RV hook recognition and positioning system based on a binocular camera, comprising: Image collection module, used to collect and create a dataset of RV hook images; The preprocessing module is used to preprocess the RV hook image dataset; A model training module, for performing model training based on the preprocessed RV hook image data set to form a first recognition model for preliminary recognition of a first range within the RV hook area, and a second recognition model for key point feature recognition within the first range; An image acquisition module is used to acquire an image to be detected and a binocular image, and process the image to be detected by using a first recognition model and a second recognition model to acquire key point feature data in the image to be detected; The data processing module is used to extract the pixel position of the key point feature data, and obtain the actual position information of the RV hook based on the pixel position information of the key point feature data and the binocular image.

[0014] By adopting the above technical solution, a RV hook image data set is first produced, the images are annotated and trained to form a first recognition model and a second recognition model. Then, the image to be detected is recognized, and the pixel position of the key point feature data in the image to be detected is obtained. Based on the pixel position information of the key point feature data and the binocular image, the actual position information of the RV hook is obtained, and the camera is used to locate the RV hook, thereby reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0016] Figure 1 The present invention is a flow chart of a method for identifying and locating a RV hook based on a binocular camera. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] Reference Figure 1 The embodiment of the present invention provides a method for identifying and locating a motorhome hook based on a binocular camera, comprising the following steps: S1: Collect and create a dataset of RV hook images.

[0019] Since there are no data samples for trailer hooks in conventional visual models, it is necessary to collect data sets for trailer hooks. The collected sample data is about several thousand, and they are collected from different vehicles, at different times, at different angles, and in different environments.

[0020] S2: Preprocess the RV hook image dataset.

[0021] The above specifically includes: Perform sample selection on the RV hook image dataset; The selection rules are set by technicians in this field and will not be described in detail here.

[0022] Annotate the images according to the requirements of the first recognition model and the second recognition model respectively; Perform data cleaning and data enhancement on the image dataset.

[0023] Among them, labeling pictures includes: When marking according to the requirements of the first recognition model, the approximate range of the hook in the picture is marked in the form of a rectangular frame, and the size of the rectangular frame is set by a person skilled in the art.

[0024] When marking according to the requirements of the second recognition model, the key points in the rectangular frame are marked. The key point in the embodiment of the present invention is the RV hook ball head.

[0025] S3: Model training is performed according to the preprocessed RV hook image data set to form a first recognition model for preliminarily identifying a first range within the RV hook area, and a second recognition model for identifying key point features within the first range.

[0026] As mentioned above, the first range is a rectangular frame where the RV hook is located, which is set by technicians in this field according to actual needs and will not be described in detail here.

[0027] Wherein, establishing the first recognition model includes: Set the data organization format to coco data format; The embodiment of the present invention adopts the transformer-based DETR target detection algorithm as the model training algorithm. The algorithm does not need to rely on manually designed components, such as the NMS non-maximum suppression algorithm. It is a direct collective prediction model that uses the transformer encoder-decoder architecture to predict all objects at one time, which is simpler and more efficient in performance.

[0028] After comparing the current mainstream visual detection algorithms, the use of the DETR algorithm can achieve higher accuracy and segmentation effects for small targets while ensuring speed. The network is mainly composed of three parts, namely the CNN backend network for feature extraction, the transformer encoder-decoder, and the feedforward network (FFN) for final detection prediction. The present invention replaces the ResNet network used in the basic backbone, and replaces the Resnet used in the backbone of the DETR network structure with MobileNetV3. Since the RV hook recognition scene is relatively fixed, the basic visible range is the RV hook behind the rear bumper of a private car. The RV hook connector structure is relatively simple and the features are relatively obvious. It does not require a backbone network with too many parameters. By replacing it with a lightweight network, the calculation can be accelerated without affecting the recognition accuracy.

[0029] The second recognition model uses a model training based on YOLOV8 for key point features.

[0030] After the first recognition model and the second recognition model are established, a quantization acceleration operation is performed on the first recognition model and the second recognition model, including: INT8 is used for model quantization, and fast forward inference of camera data is achieved by converting the pth model into an onnx model file and the final engine model file.

[0031] S4: Acquire the image to be detected and the binocular image, and process the image to be detected by using the first recognition model and the second recognition model to obtain key point feature data in the image to be detected.

[0032] Among them, since simply using the target algorithm to locate the RV hook will produce a certain position jitter, which will have a certain impact on the subsequent detection of the second recognition model, the first recognition model is used to identify the ball head. After obtaining the rectangular frame of the hook, the position of the ball head is continuously tracked and matched to achieve smooth output results. The main algorithm used in the tracking process is the detection-based sort tracking algorithm, and the modules involved are the Kalman filter algorithm and the Hungarian matching algorithm. Then the smoothed rectangular frame is cropped and the second recognition model is used to identify the rectangular frame.

[0033] S5: Extract pixel positions of key point feature data.

[0034] The above pixel positions are the pixel positions of the key point feature data in the image.

[0035] S6: According to the pixel position information of the key point feature data and the binocular image, the actual position information of the RV hook is obtained, which specifically includes: S61: calibrate the binocular camera; S62: Correcting the original binocular image according to the calibration result; As mentioned above, the binocular cameras are calibrated to obtain the internal and external parameters and homography matrix of the two cameras. The original images are corrected according to the calibration results. The two corrected images are located in the same plane and are parallel to each other, with the same Y-axis coordinates.

[0036] S63: extracting a disparity map, extracting a disparity map of corresponding pixels in the binocular image according to the detected object position information, finding the same coordinate system corresponding to the pixel coordinate systems of the left and right cameras, and calculating the difference of the same point in the X-axis direction of the pixel coordinate systems of the two cameras; Match the corresponding points between the left and right images of the binocular image based on SGBM; ; in, It represents the calculation of the neighborhood gradient around a point in the pixel coordinate system; the above formula describes the gradient calculation process, and then the result calculated by the Sobel operator is mapped by a stage function to obtain the gradient information of the image.

[0037] In calculating the cost of pixel matching, the neighborhood summation method is used, that is, the matching cost of each pixel will include information about the surrounding local area: ; in, Represents a point p in the field The sum of all the cost calculations of the surrounding neighborhood is Represents the window size, usually 3x3 or 5x5.

[0038] S65: Calculate the depth map based on the extracted disparity map, and use the binocular ranging formula to calculate the depth information of the object. The binocular ranging formula usually involves matching the pixels of the two corrected images based on the focal length, baseline length, disparity and other parameters of the camera, and calculate the depth of each pixel based on the matching results to obtain the depth map. This process is based on the principle of triangulation and includes:

[0039] ; Among them, z represents the specific depth information represented by the pixel, f represents the focal length of the camera, b represents the baseline length of the camera, and d represents the parallax, that is, the horizontal distance difference between the object on the left and right camera imaging planes.

[0040] S66: extracting the actual position information of the RV hook from the depth map in combination with the pixel position information of the key point feature data.

[0041] After obtaining the disparity depth map according to the above steps, the actual distance of the actual key point relative to the camera is extracted from the depth map in combination with the pixel coordinates of the feature points of the RV hook ball head obtained previously, thereby realizing the recognition and positioning of the RV hook.

[0042] The embodiment of the present invention provides a method for identifying and locating a motorhome hook based on a binocular camera. A data set of motorhome hook images is prepared, and the images are annotated and trained to form a first recognition model and a second recognition model. Then, the image to be detected is recognized, and the pixel position of key point feature data in the image to be detected is obtained. The actual position information of the motorhome hook is obtained based on the pixel position information of the key point feature data and the binocular image, and the camera is used to locate the motorhome hook, thereby reducing the cost.

[0043] The embodiment of the present invention also provides a RV hook identification and positioning system based on a binocular camera, comprising: Image collection module, used to collect and create a dataset of RV hook images; The preprocessing module is used to preprocess the RV hook image dataset; A model training module, for performing model training based on the preprocessed RV hook image data set to form a first recognition model for preliminary recognition of a first range within the RV hook area, and a second recognition model for key point feature recognition within the first range; An image acquisition module is used to acquire an image to be detected and a binocular image, and process the image to be detected by using a first recognition model and a second recognition model to acquire key point feature data in the image to be detected; The data processing module is used to extract the pixel position of the key point feature data, and obtain the actual position information of the RV hook based on the pixel position information of the key point feature data and the binocular image.

[0044] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0045] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0046] It can be known from the description of the above implementation modes that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various implementation modes of the present application or certain parts of the implementation modes.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying and locating a RV hook based on a binocular camera, characterized in that: include: Collect and create a dataset of RV hook images; Preprocess the RV hook image dataset; Model training is performed based on the preprocessed RV hook image data set to form a first recognition model for preliminarily identifying a first range within the RV hook area, and a second recognition model for identifying key point features within the first range; Acquire an image to be detected and a binocular image, and process the image to be detected by using a first recognition model and a second recognition model to obtain key point feature data in the image to be detected; Extract pixel positions of key point feature data; The actual position information of the RV hook is obtained based on the pixel position information of the key point feature data and the binocular image.

2. The method according to claim 1, characterized in that The key point of the RV hook is the ball head of the hook.

3. The method according to claim 1, characterized in that The preprocessing of the RV hook image dataset includes: Perform sample selection on the RV hook image dataset; Annotate the images according to the requirements of the first recognition model and the second recognition model respectively; Perform data cleaning and data enhancement on the image dataset.

4. The method according to claim 3, characterized in that The establishment of the first recognition model includes: Set the data organization format to coco data format; The transformer-based DETR target detection algorithm is used as the model training algorithm; Among them, the Resnet used in the backbone of the DETR network structure is replaced by MobileNetV3.

5. The method according to claim 4, characterized in that The second recognition model uses a model training based on YOLOV8 for key point features.

6. The method according to claim 1, characterized in that After the first recognition model and the second recognition model are established, a quantization acceleration operation is performed on the first recognition model and the second recognition model, including: INT8 is used for model quantization.

7. The method according to claim 1, characterized in that: The method of obtaining the actual position information of the RV hook according to the pixel position information of the key point feature data and the binocular image includes: Calibrate the binocular camera; Correct the original binocular image according to the calibration results; Extract disparity map; According to the extracted disparity map, a depth map is calculated; Combined with the pixel position information of the key point feature data, the actual position information of the RV hook is extracted from the depth map.

8. The method according to claim 7, characterized in that The extracting of the disparity map comprises: Match the corresponding points between the left and right images of the binocular image based on SGBM: ； in, It represents the calculation of the neighborhood gradient around a point in the pixel coordinate system; In calculating the cost of pixel matching, the neighborhood summation method is used, that is, the matching cost of each pixel will include information about the surrounding local area: ； in, Represents a point p in the field The sum of all the cost calculations of the surrounding neighborhood is Represents the window size.

9. The method according to claim 7, characterized in that: The step of calculating the depth map according to the extracted disparity map comprises: ； Among them, z represents the specific depth information represented by the pixel, f represents the focal length of the camera, b represents the baseline length of the camera, and d represents the parallax, that is, the horizontal distance difference between the object on the left and right camera imaging planes.

10. A RV hook recognition and positioning system based on a binocular camera, characterized in that: include: Image collection module, used to collect and create a dataset of RV hook images; The preprocessing module is used to preprocess the RV hook image dataset; A model training module, for performing model training based on the preprocessed RV hook image data set to form a first recognition model for preliminary recognition of a first range within the RV hook area, and a second recognition model for key point feature recognition within the first range; An image acquisition module is used to acquire an image to be detected and a binocular image, and process the image to be detected by using a first recognition model and a second recognition model to acquire key point feature data in the image to be detected; The data processing module is used to extract the pixel position of the key point feature data, and obtain the actual position information of the RV hook based on the pixel position information of the key point feature data and the binocular image.

Images