Vehicle control method, system, device and storage medium

Abstract

The present invention discloses a vehicle control method, system, device and storage medium. The method comprises: collecting images of a trailer behind a tractor to obtain an image of the trailer, then identifying a hooking mark of the trailer based on the image of the trailer to obtain the spatial coordinates of a hooking area on the trailer, and finally controlling the movement of the tractor and completing the hooking operation with the trailer based on the spatial coordinates of the hooking area. In this embodiment, the spatial coordinates of the hooking area on the trailer can be quickly and accurately identified, and then the movement of the tractor is controlled based on the spatial coordinates of the hooking area. The hooking operation with the trailer can be completed quickly and accurately without waiting for manual operation, thereby improving the efficiency of material handling.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to a vehicle control method, system, device and storage medium. Background Art

[0002] An Automated Guided Vehicle (AGV) is a highly automated device that includes a tractor and a trailer. The trailer is loaded with goods and shelves. The hook of the tractor is connected to the hook area of ​​the trailer and then moves to complete material handling. However, in the existing material handling process, the hook connection between the tractor and the trailer usually requires waiting for manual operation, which is slow and prone to manual misjudgment, resulting in the inability to quickly and accurately complete the hooking action, affecting the efficiency of material handling. Summary of the invention

[0003] The present invention provides a vehicle control method, system, device and storage medium to solve the problem that the existing hook connection between a tractor and a trailer requires manual operation, resulting in the inability to quickly and accurately complete the hook action, thus affecting the material handling efficiency.

[0004] A vehicle control method is provided, wherein the vehicle includes a tractor and a trailer, and the method includes:

[0005] Capturing an image of the trailer behind the tractor to obtain an image of the trailer;

[0006] Based on the trailer image, the trailer hook mark is identified to obtain the spatial coordinates of the hook area on the trailer;

[0007] Based on the spatial coordinates of the hooking area, the tractor is controlled to move and complete the hooking operation with the trailer.

[0008] Optionally, the trailer image includes an RGB image and a depth image, and the hook mark of the trailer is identified based on the trailer image to determine the spatial coordinates of the hook area on the trailer, including:

[0009] Recognize the hook mark of the trailer based on the RGB image to identify the hook area on the trailer;

[0010] The coordinates of the hooking area are calculated based on the depth image to obtain the spatial coordinates of the hooking area.

[0011] Optionally, the coordinates of the hooking area are calculated based on the depth image to obtain the spatial coordinates of the hooking area, including:

[0012] Identify and segment the hooking area in the depth image to obtain a depth map of the hooking area;

[0013] Based on the hook area depth map, point cloud information is converted to obtain point cloud information of the hook area;

[0014] Based on the point cloud information of the hooking area, the spatial coordinates of the hooking area are determined.

[0015] Optionally, point cloud information conversion is performed based on the hook area depth map to obtain point cloud information of the hook area, including:

[0016] Performing uniform regional segmentation on the hooking area depth map to obtain depth maps of multiple segmented areas of the hooking area;

[0017] The depth map of each segmented area is converted into point cloud information respectively to obtain point cloud information of the hooking area. The point cloud information of the hooking area includes point cloud information of multiple segmented areas.

[0018] Optionally, the depth map of each segmented area is converted into point cloud information to obtain point cloud information of the hooking area, including:

[0019] Determine the pixel value and coordinates of each pixel in the depth map of each segmented area;

[0020] Based on the imaging parameters of the image acquisition device, the pixel value and coordinates of each pixel point in the depth map of each segmented area, the depth map of each segmented area is converted into point cloud information to obtain the point cloud information of each segmented area.

[0021] Optionally, the point cloud information of the hooking area includes point cloud information of a plurality of segmented areas, and based on the point cloud information of the hooking area, determining the spatial coordinates of the hooking area includes:

[0022] Based on the point cloud information of each segmented area, determine the centroid coordinates of each segmented area;

[0023] Based on the centroid coordinates of each segmented area, the coordinates of the geometric center point of the hooking area are determined to obtain the spatial coordinates of the hooking area.

[0024] Optionally, the method further comprises

[0025] Obtain a pre-trained preset target detection model. The preset target detection model is a real-time target detection model obtained by training multiple hook marker data after improving the YOLO model based on the multi-core attention mechanism;

[0026] The hook mark of the trailer is recognized based on the RGB image to identify the hook area on the trailer, including:

[0027] The preset target detection model is used to identify the hook markers in the RGB image to identify the hook area on the trailer.

[0028] Optionally, based on the spatial coordinates of the hooking area, controlling the tractor to move and completing the hooking operation with the trailer includes:

[0029] Determine the offset angle between the hook and the hooking area on the tractor;

[0030] Based on the spatial coordinates and offset angle of the hooking area, the tractor is controlled to move and complete the hooking operation with the trailer.

[0031] Optionally, the hooking area includes a plurality of segmented areas, and determining the offset angle between the hook on the tractor and the hooking area includes:

[0032] Get the centroid coordinates of multiple segmented regions;

[0033] Based on the centroid coordinates of each segmented area and the spatial coordinates of the hooking area, the offset angle between the hook on the tractor and the hooking area is determined.

[0034] Optionally, the multiple segmented areas include an upper left area, a lower left area, an upper right area, and a lower right area, and based on the centroid coordinates of each segmented area and the spatial coordinates of the hook area, determining the offset angle between the hook on the tractor and the hook area includes:

[0035] Determine the centroid coordinates of the left area in the hook area based on the centroid coordinates of the upper left area and the centroid coordinates of the lower left area;

[0036] Determine the centroid coordinates of the right area in the hook area based on the centroid coordinates of the upper right area and the centroid coordinates of the lower right area;

[0037] Based on the coordinates of the center of mass of the left area and the coordinates of the center of mass of the right area, the offset angle between the hook on the tractor and the hook area is determined.

[0038] A vehicle control system is provided, comprising a vehicle and a vehicle control device, wherein the vehicle comprises a tractor and a trailer, and the vehicle control device is used to execute the steps of the above-mentioned vehicle control method.

[0039] An electronic device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the steps of the above-mentioned vehicle control method are implemented when the processor executes the computer instructions.

[0040] A readable storage medium is provided, wherein the readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the steps of the above-mentioned vehicle control method are implemented.

[0041] In one solution provided by the above-mentioned vehicle control method, system, device and storage medium, an image of the trailer is obtained by capturing an image of the trailer behind the tractor, and then the hooking marker of the trailer is identified based on the trailer image to obtain the spatial coordinates of the hooking area on the trailer, and finally, based on the spatial coordinates of the hooking area, the tractor is controlled to move and complete the hooking operation with the trailer; in this embodiment, the spatial coordinates of the hooking area on the trailer can be quickly and accurately identified, and then the movement of the tractor is controlled based on the spatial coordinates of the hooking area, and the hooking operation with the trailer can be completed quickly and accurately without waiting for manual operation, thereby improving the efficiency of material handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0043] Figure 1 is a schematic structural diagram of a vehicle control system in one embodiment of the present invention;

[0044] Figure 2 is a flow chart of a vehicle control method according to an embodiment of the present invention;

[0045] Figure 3 is a structural schematic diagram of a tractor and a trailer in one embodiment of the present invention;

[0046] Figure 4 yes Figure 2 A schematic diagram of an implementation process of step S20;

[0047] Figure 5 is a schematic diagram of segmentation of a hooking area depth map in one embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of the centroids of different segmented areas in one embodiment of the present invention;

[0049] Figure 7 yes Figure 2 A schematic diagram of an implementation process of step S30;

[0050] Figure 8 is a network structure diagram of a preset target detection model in one embodiment of the present invention;

[0051] Fig. 9 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or their collections. It should also be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0054] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0056] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0058] The vehicle control method provided by the embodiment of the present invention can be applied in Figure 1 In the vehicle control system shown, the vehicle control system includes a vehicle and a vehicle control device, and the vehicle includes a tractor and a trailer, wherein the tractor communicates with the vehicle control device via a bus or a network.

[0059] The vehicle control device is used to: collect images of the trailer behind the tractor to obtain an image of the trailer, then identify the hooking mark of the trailer based on the trailer image, identify the spatial coordinates of the hooking area on the trailer, and finally control the movement of the tractor and complete the hooking operation with the trailer based on the spatial coordinates of the hooking area. In this embodiment, the spatial coordinates of the hooking area on the trailer can be quickly and accurately identified, and then the movement of the tractor is controlled based on the spatial coordinates of the hooking area, and the hooking operation with the trailer can be completed quickly and accurately without waiting for manual operation, thereby improving the efficiency of material handling.

[0060] The vehicle control device may be a controller on the tractor, such as a vehicle controller. In other embodiments, the vehicle control device may also be other control devices independent of the tractor, which will not be described in detail herein.

[0061] In one embodiment, if Figure 2 As shown, a vehicle control method is provided, the vehicle includes a tractor and a trailer, and the method is applied in Figure 1 The vehicle control device in FIG. 1 is taken as an example to illustrate, and the steps are as follows:

[0062] S10: Capturing an image of the trailer behind the tractor to obtain an image of the trailer.

[0063] When it is necessary to control the tractor to hook the trailer for material handling, the vehicle control device needs to use the image acquisition device on the tractor to acquire an image of the trailer behind the tractor.

[0064] Among them, in order to facilitate the subsequent coordinate positioning of the hooking area on the trailer and reduce the amount of data processing during the coordinate positioning process to improve the timeliness of the hooking, the layout position of the image acquisition device on the tractor needs to be the same as the layout position of the hook on the tractor; or the layout position of the image acquisition device and the hook on the tractor are on the same horizontal plane.

[0065] S20: Based on the trailer image, a hook mark of the trailer is identified to obtain the spatial coordinates of the hook area on the trailer.

[0066] After acquiring the trailer image, the vehicle control device needs to identify the hook mark of the trailer based on the trailer image to identify the spatial coordinates of the hook area on the trailer. Among them, a target detection model can be obtained, and the trailer image can be input into the target detection model for hook mark recognition, so as to identify the hook area on the trailer, which is simple and convenient.

[0067] The target detection model can be a model of the YOLO (you only look once) series, such as the YOLOv5 model, the YOLOv7 model, the YOLOv8n model, etc., where v5, v7, and v8 represent version numbers, and the n in YOLOv8n represents the non-maximum suppression technology. The YOLO series model is used for hook marker recognition, which has good real-time and accuracy effects. The YOLOv8n model is preferably sampled for hook marker recognition, and the non-maximum suppression technology can be further used to optimize the multiple prediction results of the model, so that the most accurate recognition data can be output.

[0068] After identifying the hooking area on the trailer, it is necessary to determine the spatial coordinates of the hooking area on the trailer based on the trailer image. For example, the trailer image can be a depth image, and the trailer image can be converted into three-dimensional coordinates based on the depth information of the trailer image to obtain point cloud information of the trailer image, and the point cloud information includes the three-dimensional coordinates (i.e., spatial coordinates) of all pixel points (particles) in the image; then, the point cloud information of the hooking area on the trailer is determined in the point cloud information of the trailer image, so as to obtain the spatial coordinates of the hooking area, for example, the point cloud information of the geometric center point in the hooking area is used as the spatial coordinates of the hooking area.

[0069] Specifically, the three-dimensional coordinates of each pixel in the point cloud information of the trailer image are (X, Y, Z), which can be converted by the following formula:

[0070] ;

[0071] Among them, X, Y, and Z represent the coordinates of a pixel point on the x-axis, y-axis, and z-axis in the three-dimensional space coordinate system respectively; Represents the pixel value (grayscale value) of a pixel in the trailer image (depth image); Indicates the depth scale (i.e. the scale factor of the image acquisition device). The depth scale is generally set to 1000 by default. , Represents the image coordinates of a pixel in the trailer image; 、 represents the focal length of the image acquisition device on the x-axis and y-axis; , Indicates the optical center of the image acquisition device on the x-axis and y-axis.

[0072] S30: Based on the spatial coordinates of the hooking area, the tractor is controlled to move and complete the hooking operation with the trailer.

[0073] After determining the spatial coordinates of the hooking area on the trailer, the vehicle control device needs to control the tractor to move and complete the hooking operation with the trailer based on the spatial coordinates of the hooking area.

[0074] Specifically, the vehicle control device records the relative position of the image acquisition device and the hook on the tractor. The vehicle control device converts the relative position into spatial coordinates based on the current spatial coordinates of the image acquisition device to obtain the spatial coordinates of the hook on the tractor. Then, the vehicle control device plans the path of the tractor based on the spatial coordinates of the hook on the tractor and the spatial coordinates of the hooking area on the trailer, and controls the movement of the tractor according to the planned path, so that the hook on the tractor is aligned with the hooking area on the trailer, and controls the hook on the tractor to hook the hook area on the trailer to complete the hooking operation.

[0075] The structure of the tractor and trailer can be as follows Figure 3 shown. Figure 3 In the figure, the tractor 1 is equipped with an image acquisition device (which may be a depth camera) 2 and a hook 4, and the hook 4 is provided with an opening and closing device 3; the arrangement position of the image acquisition device 2 on the tractor 1 needs to be the same as the arrangement position of the hook 4 on the tractor 1, or the arrangement position of the image acquisition device 2 and the hook 4 on the tractor 1 are in the same horizontal plane. The trailer 7 is provided with wheels 8 and a hooking area 5 corresponding to the hook 4. The hooking area 5 may also be surrounded by a marker that clearly identifies the hooking area 5. The marker may be a specific pattern, such as a hooking pattern, a QR code, a barcode, and a logo. When it is necessary to control the tractor 1 to hook and connect with the rear trailer 7 for material handling, the vehicle control device acquires an image of the trailer 7 through the image acquisition device 2 on the tractor 1 to obtain an image of the trailer. Then, the vehicle control device identifies the hooking area 5 or the hooking marker 6 of the trailer 7 based on the trailer image to identify the spatial coordinates of the hooking area 5 on the trailer 7. Finally, the vehicle control device plans the path of the tractor 1 based on the spatial coordinates of the hook 4 on the tractor 1 and the spatial coordinates of the hook area 5 on the trailer 7, and controls the movement of the tractor 1 according to the planned path, so that the hook 4 on the tractor 1 is aligned with the hook area 5 on the trailer 7, and controls the opening and closing device 3 of the hook 4 on the tractor 1 to open, and then controls the hook 4 on the tractor 1 to hook the hook area 5 on the trailer 7 to complete the hooking operation. After the hooking operation of the tractor 1 and the trailer 7 is completed, the vehicle control device also needs to control the opening and closing device 3 on the hook 4 of the tractor 1 to close to prevent the hook 4 from falling off and ensure that the tractor 1 is firmly connected to the trailer 7. After that, the vehicle control device controls the movement of the tractor 1 according to the preset material handling route to achieve material handling.

[0076] In this embodiment, the image of the trailer behind the tractor is collected to obtain the trailer image, and then the hook mark of the trailer is identified based on the trailer image to obtain the spatial coordinates of the hook area on the trailer, and finally the tractor is controlled to move and the hook operation with the trailer is completed based on the spatial coordinates of the hook area. This method can quickly and accurately identify the spatial coordinates of the hook area on the trailer, and then control the movement of the tractor based on the spatial coordinates of the hook area, and the hook operation with the trailer can be completed quickly and accurately without waiting for manual operation, thereby improving the efficiency of material handling.

[0077] In one embodiment, the trailer image includes an RGB image and a depth image. Figure 4 As shown, in step S20, the hook mark of the trailer is identified based on the trailer image to determine the spatial coordinates of the hook area on the trailer, which specifically includes the following steps:

[0078] S21: Identify the hook mark of the trailer based on the RGB image to identify the hook area on the trailer.

[0079] In this embodiment, the image acquisition device may be a depth camera, which acquires an RGB image and a depth image of the trailer behind the tractor through the depth camera to obtain an RGB image and a depth image of the trailer, that is, the trailer image includes an RGB image and a depth image. R, G, and B represent the colors of the three channels of red, green, and blue, and the RGB image is a color image of the three channels of red, green, and blue.

[0080] After obtaining the RGB image and the depth image of the trailer, the vehicle control device performs hook identification on the trailer based on the RGB image to identify the hook area on the trailer. Compared with directly using the depth image to identify the hook, in this embodiment, the hook identification is performed using the color RGB image, so that the hook area on the trailer in the RGB image can be identified more timely and accurately.

[0081] S22: Calculate the coordinates of the hooking area based on the depth image to obtain the spatial coordinates of the hooking area.

[0082] After identifying the hooking area on the trailer in the RGB image, since the depth image and the RGB image are acquired by the same image acquisition device, the vehicle control device can quickly correspond to the hooking area on the trailer in the RGB image and mark it, and then perform three-dimensional coordinate conversion on the depth image based on the depth information of the depth image to obtain the point cloud information of the depth image. Therefore, based on the point cloud information of the depth image, the three-dimensional coordinates of the hooking area in the depth image are determined, and the spatial coordinates of the hooking area can be obtained.

[0083] In this embodiment, the hook marker of the trailer is identified based on the RGB image to identify the hook area on the trailer, and then the coordinates of the hook area are calculated based on the depth image to obtain the spatial coordinates of the hook area. The process of determining the spatial coordinates of the hook area on the trailer is refined, and different purposes are achieved through different types of trailer images, further improving the accuracy of hook area recognition and spatial coordinate recognition.

[0084] In one embodiment, step S22, i.e., calculating the coordinates of the hooking area based on the depth image to obtain the spatial coordinates of the hooking area, specifically includes the following steps:

[0085] S221: Identify and segment the hooking area in the depth image to obtain a hooking area depth map.

[0086] After identifying the hook area on the trailer in the RGB image, the vehicle control device can quickly correspond to the hook area on the trailer in the RGB image and mark it to obtain the hook area on the trailer in the depth image; then, the vehicle control device segments the hook area on the trailer marked in the depth image to obtain a hook area depth map.

[0087] S222: Convert point cloud information based on the hooking area depth map to obtain point cloud information of the hooking area.

[0088] After obtaining the hook area depth map, the vehicle control device needs to convert the point cloud information based on the hook area depth map to obtain the point cloud information of the hook area. The point cloud information of the hook area includes the three-dimensional coordinates of all pixel points (particles) in the hook area depth map.

[0089] Specifically, the three-dimensional coordinates of each pixel point in the point cloud information of the hooking area are converted through (X, Y, Z) by the following formula:

[0090] ;

[0091] Among them, X, Y, and Z represent the coordinates of a pixel point on the x-axis, y-axis, and z-axis in the three-dimensional space coordinate system respectively; Represents the pixel value (grayscale value) of a pixel point in the hook area depth map; Indicates the depth scale (i.e. the scale factor of the image acquisition device). The depth scale is generally set to 1000 by default. , Represents the image coordinates of a pixel point in the hook area depth map; 、 represents the focal length of the image acquisition device on the x-axis and y-axis; , Indicates the optical center of the image acquisition device on the x-axis and y-axis.

[0092] S223: Determine the spatial coordinates of the hooking area based on the point cloud information of the hooking area.

[0093] After obtaining the point cloud information of the hooking area, that is, obtaining the three-dimensional coordinates of each pixel point in the depth map of the hooking area, the vehicle control device needs to determine the spatial coordinates of the hooking area based on the point cloud information of the hooking area.

[0094] For example, the geometric center point (center of mass) of the hooking area depth map can be determined, and then the three-dimensional coordinates of the geometric center point can be used as the actual spatial coordinates of the hooking area, which is simple and convenient. The actual spatial coordinates of the hooking area can be determined by the following formula:

[0095] ;

[0096] in, Represents the geometric center point of the hook area depth map; that is Respectively The coordinates of the x-axis, y-axis and z-axis in the three-dimensional space coordinate system are the spatial coordinates of the hooking area; n represents the number of all pixels (particles) in the depth map of the hooking area, ; They respectively represent the coordinates of the pixel point i in the hook area depth map on the x-axis, y-axis and z-axis in the three-dimensional space coordinate system.

[0097] In this embodiment, the hooking area in the depth image is identified and segmented to obtain a hooking area depth map, and then point cloud information conversion is performed based on the hooking area depth map to obtain point cloud information of the hooking area. Finally, based on the point cloud information of the hooking area, the spatial coordinates of the hooking area are determined, and the specific determination process of the spatial coordinates of the hooking area is clarified. The hooking area of ​​the depth image is segmented out, and only the hooking area depth map obtained by segmentation is converted into point cloud information, which can reduce the data processing amount of point cloud information conversion and improve the real-time determination of the spatial coordinates of the hooking area.

[0098] In one embodiment, step S222, that is, performing point cloud information conversion based on the hooking area depth map to obtain point cloud information of the hooking area, specifically includes the following steps:

[0099] S2221: Evenly divide the hook area depth map into regions to obtain depth maps of multiple divided regions.

[0100] After obtaining the hooking area depth map, the vehicle control device needs to evenly divide the hooking area depth map into regions to obtain depth maps of multiple divided regions of the hooking area.

[0101] For example, the quartering method can be used to evenly divide the hook area depth map into regions, and obtain depth maps of four divided regions of the hook area. Figure 5 As shown, the hook area depth map can be segmented by the quartering method to obtain depth maps of four segmented areas 1, 2, 3, and 4.

[0102] In other embodiments, the binary method may be used to evenly divide the hooking area depth map into regions to obtain depth maps of four divided regions of the hooking area.

[0103] S2222: performing point cloud information conversion on the depth map of each segmented area respectively to obtain point cloud information of the hooking area, where the point cloud information of the hooking area includes point cloud information of multiple segmented areas.

[0104] Then, the depth map of each segmented area is converted into point cloud information to obtain point cloud information of multiple segmented areas, that is, the point cloud information of the hooking area is obtained, so that the centroid coordinates of each segmented area can be determined based on the point cloud information of each segmented area, and then the coordinates of the geometric center point of the hooking area can be determined based on the centroid coordinates of each segmented area to obtain the spatial coordinates of the hooking area.

[0105] Specifically, the depth map of each segmented area is converted into point cloud information to obtain point cloud information of multiple segmented areas, including: determining the pixel value and coordinates of each pixel in the depth map of each segmented area; based on the imaging parameters of the image acquisition device, the pixel value and coordinates of each pixel in the depth map of each segmented area, the depth map of each segmented area is converted into point cloud information to obtain point cloud information of each segmented area.

[0106] Specifically, the three-dimensional coordinates of each pixel in the point cloud information of each segmented area are converted into (X, Y, Z) by the following formula:

[0107] ;

[0108] Where X, Y, and Z represent the coordinates of a pixel point in the segmented area on the x-axis, y-axis, and z-axis in the three-dimensional space coordinate system, respectively; Represents the pixel value (grayscale value) of a pixel in the segmented area; Indicates the depth scale (i.e. the scale factor of the image acquisition device). The depth scale is generally set to 1000 by default. , Represents the image coordinates of a pixel in the segmented area; 、 represents the focal length of the image acquisition device on the x-axis and y-axis; , Indicates the optical center of the image acquisition device on the x-axis and y-axis.

[0109] In the present embodiment, the depth map of the hooking area is evenly divided into regions to obtain depth maps of multiple segmented areas of the hooking area, and the depth map of each segmented area is converted into point cloud information to obtain point cloud information of multiple segmented areas, thereby obtaining point cloud information of the hooking area. This allows the spatial coordinates of the hooking area to be determined based on the point cloud information of multiple segmented areas, providing an accurate data basis for the subsequent determination, reducing the error in obtaining the geometric center through a single point cloud information, and improving the accuracy of the spatial coordinates of the hooking area.

[0110] In one embodiment, the point cloud information of the hooking area includes point cloud information of multiple segmented areas. In step S223, point cloud information conversion is performed based on the hooking area depth map to obtain point cloud information of the hooking area, which specifically includes the following steps:

[0111] S2231: Determine the centroid coordinates of each segmented area based on the point cloud information of each segmented area.

[0112] In this embodiment, the point cloud information of the hooking area includes point cloud information of multiple segmented areas. The vehicle control device needs to determine the geometric center coordinates of each segmented area based on the point cloud information of each segmented area, that is, determine the centroid coordinates of each segmented area. The centroid coordinates of each segmented area are calculated as follows:

[0113] Among them, the centroid coordinates of each segmented area It can be determined by the following formula:

[0114] ;

[0115] in, Represents the geometric center point (centroid) of a segmented area; that is Respectively The coordinates of the x-axis, y-axis, and z-axis in the three-dimensional space coordinate system are the coordinates of the centroid of a segmented area; n represents the number of all pixels (particles) in a segmented area. ; They respectively represent the coordinates of the pixel point i in a segmented area on the x-axis, y-axis, and z-axis in the three-dimensional space coordinate system.

[0116] S2232: Based on the centroid coordinates of each segmented area, determine the coordinates of the geometric center point of the hooking area to obtain the spatial coordinates of the hooking area.

[0117] Then, the vehicle control device determines the coordinates of the geometric center point of the hooking area based on the centroid coordinates of each divided area, and obtains the spatial coordinates of the hooking area.

[0118] In one embodiment, if the point cloud information of the hooking area includes the point cloud information of two segmented areas, the point cloud information of the two segmented areas may be the point cloud information of the left area (or the upper area) and the point cloud information of the right area (or the lower area); based on the point cloud information of the two segmented areas, the centroid coordinates of the two segmented areas may be determined respectively, and then the geometric center coordinates of the hooking area may be determined based on the centroid coordinates of the two segmented areas, so as to obtain the spatial coordinates of the hooking area. Among them, the centroid coordinates of the two segmented areas may be added together, and the average value of the added coordinates may be taken to obtain the geometric center coordinates of the hooking area, and the calculation formula thereof is as follows:

[0119] ;

[0120] in, Indicates the geometric center coordinates of the hooking area, that is, the spatial coordinates of the hooking area; , They respectively represent the centroid coordinates of the left area (or upper area) and the centroid coordinates of the right area (or lower area).

[0121] In one embodiment, the point cloud information of the hooking area includes the point cloud information of the four segmented areas. Based on the point cloud information of the four segmented areas, the centroid coordinates of the four segmented areas can be correspondingly determined, and then based on the centroid coordinates of the four segmented areas, the centroid coordinates of the left area and the centroid coordinates of the right area in the hooking area can be determined. Then, after adding the centroid coordinates of the left area and the centroid coordinates of the right area, the average value of the added coordinates is taken to obtain the geometric center coordinates of the hooking area.

[0122] like Figure 5 As shown, the depth map of the hooking area can be segmented by the quartering method to obtain the depth maps of the four segmented areas 1, 2, 3, and 4, and then the depth maps of the four segmented areas 1, 2, 3, and 4 are converted into point cloud information to obtain the point cloud information of the hooking area. The point cloud information of the hooking area includes the point cloud maps (point cloud information) of the four segmented areas 1, 2, 3, and 4, as shown in Figure 6 Then, based on the point cloud information of the four segmented areas 1, 2, 3, and 4, the centroid coordinates of the four segmented areas are determined: P1, P2, P3, and P4. Then, based on the centroid coordinates of the four segmented areas P1, P2, P3, and P4, the centroid coordinates a of the left area and the centroid coordinates b of the right area in the hook area are determined. Then, after adding the centroid coordinates a of the left area and the centroid coordinates b of the right area, the average value of the added coordinates is taken to obtain the geometric center coordinates c of the hook area, where the spatial coordinates of the hook area are calculated as follows:

[0123] ;

[0124] ;

[0125] ;

[0126] in, Indicates the geometric center coordinates of the hooking area, that is, the spatial coordinates of the hooking area; , They represent the centroid coordinates of the left area and the right area respectively; P1, P2, P3, and P4 represent the centroid coordinates of the four segmented areas respectively.

[0127] In other embodiments, the centroid coordinates of the upper area in the hooking area and the centroid coordinates of the upper area can also be determined based on the centroid coordinates of the four segmented areas P1, P2, P3, and P4, and then the two centroid coordinates are added together and the average value of the added coordinates is taken to obtain the geometric center coordinates of the hooking area. The specific calculation process is referred to the previous text and will not be repeated here.

[0128] The depth map of the hooking area is divided into four parts by the quartering method, and then the centroid coordinates of the four divided areas are determined based on the point cloud information of the four divided areas, and then the spatial coordinates of the hooking area are determined. This can reduce the error caused by obtaining the geometric center through a single point cloud information. Center positioning through the four centroid coordinates can further improve the accuracy of the spatial coordinates of the hooking area.

[0129] In this embodiment, based on the point cloud information of each segmented area, the centroid coordinates of each segmented area are determined, and then based on the centroid coordinates of each segmented area, the geometric center point coordinates of the hooking area are determined to obtain the spatial coordinates of the hooking area. This can reduce the error caused by calculating the geometric center through a single point cloud information, and can improve the accuracy of the spatial coordinates of the hooking area.

[0130] In one embodiment, if Figure 7 As shown, in step S30, based on the spatial coordinates of the hooking area, the tractor is controlled to move and the hooking operation with the trailer is completed, which specifically includes the following steps:

[0131] S31: Determine an offset angle between the hook on the tractor and the hooking area.

[0132] Since the parking positions of the tractor and trailer cannot be completely accurate, the hook area (or the center of the hook area) of the tractor and the trailer cannot be completely aligned, and there will be a certain offset angle, such as left or right offset. Figure 6 As shown, with the coordinates of points a and b forming a horizontal axis parallel to the ground, the hook of the tractor and the hook area (or the center of the hook area) of the trailer cannot be completely aligned, and there is a deviation to the left or right. In this case, there will be an offset angle θ between the hook on the tractor and the hook area of ​​the trailer.

[0133] In order to ensure the accuracy of subsequent tractor movement control and avoid collision due to the hook and the towing area missing each other during the movement of the tractor, the vehicle control area also needs to determine the offset angle between the hook on the tractor and the hook area based on the spatial coordinates of the hook area.

[0134] For example, the offset angle between the hook on the tractor and the hook area can be determined by using the traditional normal vector estimation method, that is, the normal vector of the point cloud map (point cloud information) of the hook area is determined, and the offset angle between the hook on the tractor and the hook area is determined by observing the normal vector in the point cloud map.

[0135] S32: Based on the spatial coordinates and the offset angle of the hooking area, the tractor is controlled to move and complete the hooking operation with the trailer.

[0136] After determining the offset angle between the hook on the tractor and the hooking area, the vehicle control device needs to control the movement of the tractor and complete the hooking operation with the trailer based on the spatial coordinates of the hooking area and the offset angle.

[0137] For example, the vehicle control device first controls the movement of the tractor according to the offset angle, such as moving a certain angle to the left or right, so that the hook of the tractor is aligned with the hook area of ​​the trailer, and then plans the path of the tractor directly based on the spatial coordinates of the hook area on the trailer, and controls the movement of the tractor according to the planned path, and controls the opening and closing device of the hook on the tractor to open, until the tractor is controlled to hook the hook area on the trailer to complete the hooking operation, and then controls the opening and closing device on the tractor hook to close to prevent the hook from falling off, thereby ensuring a firm connection between the tractor and the trailer.

[0138] In this embodiment, based on determining the offset angle between the hook and the hooking area on the tractor, and based on the spatial coordinates and the offset angle of the hooking area, the movement of the tractor is controlled and the hooking operation with the trailer is completed. The spatial coordinates and the offset angle of the hooking area can be determined quickly and accurately, and then the movement of the tractor is controlled based on the spatial coordinates and the offset angle of the hooking area. The control accuracy of the tractor can be improved, and then the movement of the tractor can be controlled more accurately and quickly to complete the hooking operation with the trailer, further improving the efficiency of material handling.

[0139] In one embodiment, the hooking area includes a plurality of segmented areas. In step S31, the offset angle between the hook on the tractor and the hooking area is determined, which specifically includes the following steps:

[0140] S311: Obtaining centroid coordinates of multiple segmented regions;

[0141] S312: Determine an offset angle between the hook on the tractor and the hooking area based on the centroid coordinates of the segmented area.

[0142] The hooking area may include multiple segmented areas. After determining the spatial coordinates of the hooking area, the centroid coordinates of the multiple segmented areas can be obtained first. The centroid coordinates of each segmented area are obtained by segmenting the hooking area and then calculating the geometric center coordinates of each segmented area. The specific acquisition process is described above and will not be repeated here.

[0143] Then, based on the centroid coordinates of the segmented areas and the spatial coordinates of the hooking area, the offset angle between the hook on the tractor and the hooking area is determined. The centroid coordinates of the left area and the right area in the hooking area can be determined based on the centroid coordinates of the multiple segmented areas, and then based on the centroid coordinates of the left area and the right area in the hooking area, a geometric calculation is performed to obtain the offset angle between the hook on the tractor and the hooking area.

[0144] If the multiple segmented areas include an upper left area, a lower left area, an upper right area, and a lower right area, then based on the centroid coordinates of each segmented area and the spatial coordinates of the hook area, the offset angle between the hook on the tractor and the hook area is determined, specifically including the following steps: based on the centroid coordinates of the upper left area and the centroid coordinates of the lower left area, the centroid coordinates of the left area in the hook area are determined; based on the centroid coordinates of the upper right area and the centroid coordinates of the lower right area, the centroid coordinates of the right area in the hook area are determined; based on the centroid coordinates of the left area and the centroid coordinates of the right area, the offset angle between the hook on the tractor and the hook area is determined. The process of determining each centroid coordinate is as described above and will not be repeated here.

[0145] Specifically, the offset angle between the hook on the tractor and the hook area can be calculated by the following formula:

[0146] ;

[0147] in, Indicates the offset angle between the hook and the hooking area on the tractor; They respectively represent the centroid coordinates of the left area and the right area in the hook area, and the centroid coordinates all include three-dimensional coordinates (X, Y, Z), where X, Y, and Z respectively represent the coordinates of the x-axis, y-axis, and z-axis in the three-dimensional space coordinate system; Indicates a reference identifier. It means that only the Z coordinate in the centroid coordinate is referenced. Indicates that only the X coordinate in the centroid coordinates is referenced.

[0148] In this embodiment, by obtaining the centroid coordinates of multiple segmented areas, and then determining the offset angle between the hook on the tractor and the hook area based on the centroid coordinates of each segmented area and the spatial coordinates of the hook area, the offset angle can be determined quickly and accurately. Compared with the offset angle calculation using the normal vector estimation method, this method does not have the problem of excessive angle offset caused by excessive deviation in the normal vector direction, thereby improving the accuracy of the offset angle, and thus can more accurately and quickly control the movement of the tractor to complete the hooking operation with the trailer, further improving the efficiency of material handling.

[0149] In one embodiment, before the hook mark of the trailer is recognized based on the RGB image, a pre-trained preset target detection model needs to be obtained first, and the pre-trained preset target detection model is used to recognize the hook mark of the RGB image to identify the hook area on the trailer. The preset target detection model is a real-time target detection model obtained by training multiple hook mark data after improving the YOLO model based on the multi-core attention mechanism.

[0150] Among them, in step S21, the hook mark of the trailer is recognized based on the RGB image to identify the hook area on the trailer, including the following steps:

[0151] S211: Using a preset target detection model to perform hook identification on the RGB image to identify the hook area on the trailer.

[0152] When identifying the hook mark of the trailer, the vehicle control device needs to first obtain a pre-trained preset target detection model. The preset target detection model is a real-time target detection model obtained by training multiple hook mark data after improving the YOLO model based on the multi-core attention mechanism. Then, the vehicle control device inputs the RGB image into the preset target detection model, and classifies and identifies the hook mark through the preset target detection model to identify the hook area on the trailer in the RGB image, which is simple and fast, and has higher recognition accuracy.

[0153] Specifically, this embodiment uses the YOLOv8n model as the benchmark model, and the preset target detection model is a real-time target detection model obtained by training multiple hook marker data after improving the YOLOv8n model based on the multi-core attention mechanism. Figure 8 As shown, the preset target detection model includes a backbone network, a neck network and a head network. The head network includes a first detection head Detect1, a second detection head Detect2 and a third detection head Detect3. The neck network is an improved network of the neck network of YOLO based on a multi-kernel attention mechanism, that is, the neck network includes a multi-kernel attention processor (Multi-kernel Attention).

[0154] In this embodiment, the pre-trained preset target detection model is used to identify the hook marker on the RGB image to identify the hook area on the trailer, which is simple, fast and highly accurate.

[0155] In one embodiment, in step 211, a hook marker is identified on the RGB image using a preset target detection model to identify a hook area on the trailer, which specifically includes the following steps:

[0156] S201: Input the RGB image into a preset target detection model, perform multi-scale convolution feature processing through the backbone network, and obtain the first feature, the second feature, and the third feature of the RGB image input.

[0157] The backbone network includes the first feature layer, the second feature layer and the third feature layer connected in sequence, each of which includes a convolution module Conv and an optimization module C2f (coarse to fine), and the third feature layer adds a spatial pyramid pooling module SPPF (spatial pyramid pooling-fast). Figure 8 As shown, the first feature layer includes two convolution modules Conv, a C2f module, a convolution module Conv and a C2f module connected in sequence. The second feature layer includes a convolution module Conv and a C2f module connected in sequence, and the third feature layer includes a convolution module Conv, a C2f module and an SPPF module connected in sequence.

[0158] Specifically, the backbone network performs multi-scale convolution feature processing on the RGB image, including: inputting the RGB image into a preset target detection model, performing feature processing on the RGB image through the first feature layer of the backbone network, and obtaining the first feature output by the first feature layer; performing feature processing on the first feature output by the second feature layer, and obtaining the second feature output by the second feature layer; performing feature processing on the second feature output by the third feature layer, and obtaining the third feature output by the third feature layer. Among them, the feature processing in the first feature layer includes: convolution of two convolution modules Conv, feature optimization of the C2f module, re-convolution of the convolution module Conv, and re-optimization of the C2f module. The feature processing in the second feature layer includes re-convolution of the convolution module Conv and re-optimization of the feature of the C2f module. The feature processing in the second feature layer includes re-convolution of the convolution module Conv, re-optimization of the feature of the C2f module, and fast spatial pyramid pooling of the SPPF module.

[0159] In this embodiment, multiple feature layers of the backbone network include a convolution module Conv and a C2f module, wherein the third feature layer further adds a spatial pyramid pooling module SPPF, which can perform multi-scale feature extraction through the backbone network, thereby improving feature diversity and the real-time performance of feature extraction, and providing a solid foundation for subsequent multi-scale-based feature prediction.

[0160] S202: Perform multi-core feature fusion processing on the first feature, the second feature, and the third feature through the neck network to obtain a first fusion feature, a second fusion feature, and a third fusion feature.

[0161] Then, the neck network performs multi-core feature fusion processing based on the multi-core attention mechanism on the first feature, the second feature and the third feature to obtain the first fused feature, the second fused feature and the third fused feature.

[0162] For example, the neck network of the traditional YOLOvn8 model includes a first fusion layer, a second fusion layer, and a third fusion layer. A multi-core attention processor can be directly added to the neck network of the traditional YOLOvn8 model, and the multi-core attention processor is connected to multiple fusion layers. The first feature, the second feature, and the third feature can be directly input into the multi-core attention processor, and at least two of the first feature, the second feature, and the third feature are fused by the multi-core attention processor to obtain a multi-core fusion feature; then the multi-core fusion feature is combined with the first feature, the second feature, and the third feature respectively to obtain three feature groups, and the three feature groups are respectively input into the first fusion layer, the second fusion layer, and the third fusion layer to perform feature fusion processing respectively, to obtain a first fusion feature output by the first fusion layer, a second fusion feature output by the second fusion layer, and a third fusion feature output by the third fusion layer.

[0163] S203: Predict the first fusion feature, the second fusion feature, and the third fusion feature through the head network to obtain the hooking area on the trailer.

[0164] After obtaining the first fusion feature, the second fusion feature and the third fusion feature, the first fusion feature, the second fusion feature and the third fusion feature are respectively identified and predicted through the head network to obtain the identification data output by the head network, that is, the hooking area on the trailer.

[0165] Specifically, the first detection head Detect1, the second detection head Detect2 and the third detection head of the head network respectively detect the first fusion feature, the second fusion feature and the third fusion feature, that is, the first detection head Detect1 detects and identifies the first fusion feature, the second detection head Detect2 detects and identifies the second fusion feature, and the third detection head Detect3 detects and identifies the third fusion feature, thereby obtaining the detection data of the first detection head Detect1, the second detection head Detect2 and the third detection head, and then performing non-maximum suppression on the three detection data based on the non-maximum suppression technology, thereby obtaining the optimal detection data output and obtaining the final recognition data.

[0166] In this embodiment, the RGB image is input into the preset target detection model, and multi-scale convolution feature processing is performed through the backbone network to obtain the first feature, the second feature and the third feature of the RGB image input, and then the first feature, the second feature and the third feature are subjected to multi-core feature fusion processing through the neck network to obtain the first fusion feature, the second fusion feature and the third fusion feature, and finally the first fusion feature, the second fusion feature and the third fusion feature are subjected to multi-scale feature prediction through the head network to obtain the hooking area on the trailer, which clarifies the prediction process of the preset target detection model, and optimizes the neck network of the YOLOv8n model based on the multi-core attention mechanism, so as to more accurately fuse the multi-scale fusion features, and then more accurately identify the hooking area, which provides an effective data basis for the subsequent coordinate positioning of the hooking area.

[0167] In one embodiment, the neck network includes a multi-core attention feature layer, and a first fusion layer, a second fusion layer, and a third fusion layer connected in sequence. Specifically, in step S203, the neck network performs multi-core feature fusion processing on the first feature, the second feature, and the third feature, including:

[0168] S2021: Perform multi-core attention fusion processing on the first feature, the second feature and the third feature through the multi-core attention feature layer to obtain a multi-core fusion feature.

[0169] like Figure 8As shown, the multi-kernel attention feature layer includes a sampling module Upsample, a multi-kernel attention processor Multi-kernel Attention and a multi-kernel fusion device. Among them, the multi-kernel fusion device includes a convolution module Conv and a C2f module connected in sequence. Among them, the multi-kernel attention fusion processing of the second feature and the third feature through the multi-kernel attention feature layer includes the following steps: inputting the third feature into the sampling module of the multi-kernel attention feature layer for upsampling to obtain the first sampling feature; inputting the second feature into the multi-kernel attention processor for multi-kernel attention processing to obtain the multi-kernel attention feature; inputting the first sampling feature and the multi-kernel attention feature into the multi-kernel fusion device for fusion processing to obtain the multi-kernel fusion feature. By sampling the third feature and combining it with the second feature, a more accurate multi-kernel fusion feature can be obtained, so that multiple feature fusion layers can be fused based on the multi-kernel fusion feature in the future, and two multi-kernel feature fusions can be realized, which can improve the accuracy of the features output by each subsequent feature fusion layer.

[0170] In other embodiments, the first feature (or the first feature and the second feature after being concatenated) can also be input into the sampling module of the multi-core attention feature layer for upsampling to obtain the first sampling feature, so that the first sampling feature and the multi-core attention feature can be subsequently input into the multi-core fuser for fusion processing to obtain the multi-core fusion feature.

[0171] The multi-kernel attention processor Multi-kernel Attention includes multiple depth-separable convolution modules DWConv with different convolution kernels, as well as multiple channel attention modules, normalization modules BatchNorm and spatial attention modules SpatialAttention corresponding to each depth-separable convolution module. Figure 8 As shown, the multiple depth-wise separable convolution modules with different convolution kernels include depth-wise separable convolution module 1, depth-wise separable convolution module 2, and depth-wise separable convolution module 3, and the three depth-wise separable convolution modules with convolution kernels of 7, 9, and 11 respectively; Figure 8 The + symbol in represents addition.

[0172] The second feature is input into the multi-core attention processor for multi-core attention processing to obtain a multi-core attention feature, including the following steps: the second feature is input into the depth separable convolution module for depth separable convolution to obtain multiple depth separable convolution features; the depth separable convolution features output by the corresponding depth separable convolution module are processed by each channel attention module to obtain multiple channel attention features, and the attention features of each channel are added to obtain the initial attention feature; the initial attention feature is input into the normalization module for normalization processing to obtain a normalized feature, and the normalized feature is processed by the spatial attention module to obtain a multi-core attention feature. By performing deep separable convolution of multiple convolution kernels on the second feature, and then performing attention feature processing of multiple channels, and finally performing the spatial attention module after normalization, the accuracy of the output feature can be effectively improved, and the feature dimension can be reduced, and the feature processing efficiency can be improved.

[0173] In other embodiments, the number and convolution kernels of the multiple depthwise separable convolution modules DWConv may be other, which will not be elaborated here.

[0174] like Figure 8 As shown, each channel attention module ChannelAttention can include a global average pooling module Avgpool, a convolution module Conv, and an activation module Sigmoid connected in sequence. The depth-separable convolution features output by the corresponding depth-separable convolution module are processed by each channel attention module, including: firstly performing global average pooling on the input depth-separable convolution features by the global average pooling module, then convolving the depth-separable convolution features after global average pooling by the convolution module, and then performing Sigmoid function activation processing on the depth-separable convolution features after convolution by the activation module to obtain the weight value of the depth-separable convolution features; then multiplying the depth-separable convolution features with their weights to obtain the channel attention features after activation, thereby obtaining the channel attention features output by multiple channel attention modules. The weight values ​​are obtained by global average pooling, convolution, and activation processing, and then weight activation processing is performed, which improves the expression ability of the channel attention features, thereby improving the accuracy of the channel attention features.

[0175] In other embodiments, each channel attention module may also have other structural components. For example, it may include a global average pooling module Avgpool or a convolution module Conv, and an activation module Sigmoid. In addition, the activation module may also be an activation module that uses other activation functions, such as a Maxmoid function.

[0176] like Figure 8As shown, the spatial attention module SpatialAttention may include a maximum pooling module Maxpool, a global average pooling module Avgpool, a concatenation module Concat, a convolution module Conv, and an activation module Sigmoid. Among them, the normalized features are processed by the spatial attention module to obtain multi-core attention features, including: performing maximum pooling on the normalized features through the maximum pooling module to obtain the maximum pooling features, and performing global average pooling on the normalized features through the global average pooling module to obtain the average pooling features; then concatenating the maximum pooling features and the average pooling features through the average module to obtain the concatenated features, and convolving the concatenated features through the convolution module; then activating the concatenated features after convolution with the Sigmoid function through the activation module to obtain the weight value of the normalized features; finally, the weight value of the normalized features is multiplied by the normalized features to obtain the multi-core attention features after activation. The weight values ​​are obtained through a variety of average pooling, convolution and activation processing, and then weight activation processing is performed, which further improves the expression ability of the multi-core attention features, thereby further improving the accuracy of the multi-core attention features.

[0177] In other embodiments, each spatial attention module may also have other structural components. For example, the activation module may also be an activation module using other activation functions.

[0178] S2022: Fusing the first feature and the multi-core fusion feature through a first fusion layer to obtain a first fusion feature.

[0179] After obtaining the multi-core fusion feature, it is necessary to fuse the first feature and the multi-core fusion feature through a first fusion layer to obtain a first fusion feature.

[0180] Specifically, Figure 8 As shown, the first fusion layer includes a sampling module Upsample, a concatenation module Concat, and an optimization module C2f. The first feature and the multi-core fusion feature are fused, including: upsampling the multi-core fusion feature through the sampling module Upsample to obtain a second sampling feature; concatenating the first feature and the second sampling feature through the concatenation module Concat of the first fusion layer, and inputting the concatenated feature into the optimization module C2f for feature optimization to obtain the first fusion feature. Upsampling the multi-core fusion feature and then concatenating and optimizing it with the first feature can improve the feature information of the first fusion feature and improve the accuracy of the first fusion feature.

[0181] S2023: Fusing the first fusion feature and the multi-core fusion feature through a second fusion layer to obtain a second fusion feature.

[0182] After obtaining the first fusion feature, it is necessary to fuse the first fusion feature and the multi-core fusion feature through a second fusion layer to obtain a second fusion feature.

[0183] Specifically, Figure 8 As shown, the second fusion layer includes a convolution module Conv, a splicing module Concat and an optimization module C2f. The first fusion feature and the multi-core fusion feature are fused, including: the first fusion feature is convolved by the convolution module Conv of the second fusion layer to obtain the first convolution feature; then the first convolution feature and the multi-core fusion feature are spliced ​​by the splicing module Concat of the second fusion layer, and the spliced ​​features are input into the optimization module C2f for feature optimization to obtain the second fusion feature. By fusing and optimizing the first fusion feature and the multi-core fusion feature, the feature information of the second fusion feature can be further improved, thereby further improving the accuracy of the second fusion feature.

[0184] S2024: Fusing the second fusion feature and the third feature through a second fusion layer to obtain a third fusion feature.

[0185] After obtaining the second fusion feature, the second fusion feature and the third feature are input into the second fusion layer for fusion processing to obtain the third fusion feature.

[0186] Specifically, Figure 8 As shown, the third fusion layer includes a convolution module Conv, a concatenation module Concat, and an optimization module C2f; the second fusion feature and the third feature are fused through the second fusion layer, including: the second fusion feature is convolved through the convolution module Conv of the third fusion layer to obtain the second convolution feature; the second convolution feature and the third feature are concatenated through the concatenation module Concat of the third fusion layer, and the concatenated features are input into the optimization module C2f for feature optimization to obtain the third fusion feature. The feature information of the third fusion feature can be further improved, thereby further improving the accuracy of the third fusion feature.

[0187] In this embodiment, a multi-core attention feature layer is used to perform multi-core attention fusion processing on the first feature, the second feature, and the third feature input to obtain a multi-core fusion feature, and the first feature and the multi-core fusion feature are fused through the first fusion layer to obtain a first fusion feature, and then the first fusion feature and the multi-core fusion feature are fused through the second fusion layer to obtain a second fusion feature, and then the second fusion feature and the third feature are fused through the second fusion layer to obtain a third fusion feature, which clarifies the structure of the neck network and the specific acquisition process of multiple fusion features. This process determines multiple fusion features based on the multi-core attention mechanism, which can improve the accuracy of each fusion feature, and thus provide the accuracy of subsequent predictions.

[0188] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0189] In one embodiment, a vehicle control device is provided, which may be an electronic device on a vehicle. The vehicle control device corresponds one-to-one with the vehicle control method in the above embodiment. The vehicle control device includes a collection module, an identification module and a control module. Each functional module is described in detail as follows:

[0190] The acquisition module is used to acquire images of the trailer behind the tractor to obtain an image of the trailer;

[0191] A recognition module, used to recognize the hook mark of the trailer based on the trailer image, and obtain the spatial coordinates of the hook area on the trailer;

[0192] The control module is used to control the movement of the tractor and complete the hooking operation with the trailer based on the spatial coordinates of the hooking area.

[0193] Optionally, the trailer image includes an RGB image and a depth image, and the recognition module is specifically used for:

[0194] Recognize the hook mark of the trailer based on the RGB image to identify the hook area on the trailer;

[0195] The coordinates of the hooking area are calculated based on the depth image to obtain the spatial coordinates of the hooking area.

[0196] Optionally, the identification module is further specifically used for:

[0197] Identify and segment the hooking area in the depth image to obtain a depth map of the hooking area;

[0198] Based on the depth map of the hooking area, point cloud information is converted to obtain point cloud information of the hooking area;

[0199] Based on the point cloud information of the hooking area, the spatial coordinates of the hooking area are determined.

[0200] Optionally, the identification module is further specifically used for:

[0201] Performing uniform regional segmentation on the hooking area depth map to obtain depth maps of multiple segmented areas of the hooking area;

[0202] The depth map of each segmented area is converted into point cloud information respectively to obtain point cloud information of the hooking area, where the point cloud information of the hooking area includes point cloud information of multiple segmented areas;

[0203] Based on the point cloud information of each segmented area, determine the centroid coordinates of each segmented area;

[0204] Based on the centroid coordinates of each segmented area, the coordinates of the geometric center point of the hooking area are determined to obtain the spatial coordinates of the hooking area.

[0205] Optionally, the control module is specifically used for:

[0206] Determine the offset angle between the hook and the hooking area on the tractor;

[0207] Based on the spatial coordinates and offset angle of the hooking area, the tractor is controlled to move and complete the hooking operation with the trailer.

[0208] For the specific definition of the vehicle control device, please refer to the definition of the vehicle control method above, which will not be repeated here. Each module in the above-mentioned vehicle control device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0209] In one embodiment, an electronic device is provided, which may be a vehicle controller. The electronic device includes a processor, a memory, and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer instructions in the non-volatile storage medium. When the computer instructions are executed by the processor, a vehicle control method is implemented. The database of the electronic device is used to store data generated by the above-mentioned vehicle control method.

[0210] In one embodiment, Fig. 9 As shown, an electronic device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, and the steps of the above-mentioned vehicle control method are implemented when the processor executes the computer instructions.

[0211] In one embodiment, a readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the above-mentioned vehicle control method are implemented.

[0212] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer instructions, and the computer instructions can be stored in a non-volatile computer-readable storage medium. When the computer instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0213] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0214] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A vehicle control method, characterized in that: The vehicle includes a tractor and a trailer, and the method includes: Capturing an image of the trailer behind the tractor to obtain an image of the trailer; Based on the trailer image, a hook mark of the trailer is identified to obtain the spatial coordinates of the hook area on the trailer; Based on the spatial coordinates of the hooking area, controlling the tractor to move and complete the hooking operation with the trailer; The trailer image includes an RGB image and a depth image, and the identifying the hook mark of the trailer based on the trailer image to determine the spatial coordinates of the hook area on the trailer includes: Performing hook identification on the trailer based on the RGB image to identify a hooking area on the trailer, wherein the hooking area on the trailer is obtained by performing hook identification on the RGB image based on a preset target detection model; Calculating the coordinates of the hooking area based on the depth image to obtain the spatial coordinates of the hooking area; The hooking area includes a plurality of segmented areas, and based on the spatial coordinates of the hooking area, the tractor is controlled to move and complete the hooking operation with the trailer, including: Obtaining centroid coordinates of a plurality of segmented regions; Determine the offset angle between the hook on the tractor and the hook area based on the centroid coordinates of each segmented area; Based on the spatial coordinates of the hooking area and the offset angle, the tractor is controlled to move and complete the hooking operation with the trailer.

2. The vehicle control method according to claim 1, characterized in that: The calculating the coordinates of the hooking area based on the depth image to obtain the spatial coordinates of the hooking area includes: Identify and segment the hooking area in the depth image to obtain a hooking area depth map; Performing point cloud information conversion based on the hooking area depth map to obtain point cloud information of the hooking area; Based on the point cloud information of the hooking area, the spatial coordinates of the hooking area are determined.

3. The vehicle control method according to claim 2, characterized in that: The step of converting point cloud information based on the hooking area depth map to obtain point cloud information of the hooking area includes: Performing uniform regional segmentation on the hooking area depth map to obtain depth maps of multiple segmented areas of the hooking area; The depth map of each of the segmented areas is converted into point cloud information to obtain point cloud information of the hooking area, and the point cloud information of the hooking area includes point cloud information of multiple segmented areas.

4. The vehicle control method according to claim 3, characterized in that: The step of converting the depth map of each segmented area into point cloud information to obtain point cloud information of the hooking area includes: Determine the pixel value and coordinates of each pixel point in the depth map of each segmented area; Based on the imaging parameters of the image acquisition device, the pixel value and coordinates of each pixel point in the depth map of each segmented area, the depth map of each segmented area is converted into point cloud information to obtain the point cloud information of each segmented area.

5. The vehicle control method according to claim 2, characterized in that: The point cloud information of the hooking area includes point cloud information of a plurality of segmented areas, and determining the spatial coordinates of the hooking area based on the point cloud information of the hooking area includes: Determining the centroid coordinates of each segmented area based on the point cloud information of each segmented area; Based on the centroid coordinates of each segmented area, the coordinates of the geometric center point of the hooking area are determined to obtain the spatial coordinates of the hooking area.

6. The vehicle control method according to claim 1, characterized in that: Also includes: Obtain a pre-trained preset target detection model, where the preset target detection model is a real-time target detection model obtained by training a YOLO model based on a multi-core attention mechanism and using multiple hook marker data; The performing hook identification on the trailer based on the RGB image to identify the hook area on the trailer includes: The preset target detection model is used to perform hook identification on the RGB image to identify the hook area on the trailer.

7. The vehicle control method according to claim 1, characterized in that: The plurality of segmented areas include an upper left area, a lower left area, an upper right area, and a lower right area. The determining of the offset angle between the hook on the tractor and the hook area based on the centroid coordinates of each segmented area includes: Determine the centroid coordinates of the left area in the hooking area based on the centroid coordinates of the upper left area and the centroid coordinates of the lower left area; Determine the centroid coordinates of the right area in the hooking area based on the centroid coordinates of the upper right area and the centroid coordinates of the lower right area; Based on the centroid coordinates of the left area and the centroid coordinates of the right area, an offset angle between the hook on the tractor and the hooking area is determined.

8. A vehicle control system, characterized in that: The method comprises a vehicle and a vehicle control device, wherein the vehicle comprises a tractor and a trailer, and the vehicle control device is used to execute the steps of the vehicle control method according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer instructions, the steps of the vehicle control method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the vehicle control method according to any one of claims 1 to 7 are implemented.

Images