Two-vehicle distance measurement method, distance measurement device, system and computer-readable storage medium

Abstract

The present invention provides a two-vehicle ranging method, ranging device, system, and computer-readable storage medium, belonging to the field of digital image processing. The two-vehicle ranging method is applied to the ranging device, wherein the ranging device is equipped with a binocular camera. The ranging device is arranged on a first vehicle, and the first vehicle and the second vehicle cooperate with each other. The method comprises: acquiring a character image of the second vehicle captured by the binocular camera in real time, thereby obtaining a point cloud image of the character image, and then calculating the center coordinates of the point cloud image. The distance between the first vehicle and the second vehicle is calculated based on the center coordinates, so as to improve the problem that existing target detection methods cannot accurately determine whether the distance between the two vehicles is appropriate, resulting in poor control effect.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a two-vehicle distance measurement method, distance measurement equipment, system and computer-readable storage medium. Background Art

[0002] Image processing is a crucial component of autonomous driving technology. Autonomous vehicles use image processing results to perform actions such as control and event handling. Object detection, a crucial step in image processing, aims to identify all objects of interest in an image and determine their location and size. Currently, binocular camera-based object detection and positioning methods include traditional object detection and deep learning-based object detection.

[0003] Traditional target detection methods are divided into three parts: region selection, feature extraction, and classifiers. Deep learning target detection methods fall into two categories: two-stage and one-stage. However, in unmanned agricultural production, two or more unmanned vehicles often work together, such as harvesters and grain transfer vehicles. Existing target detection methods and GPS positioning methods cannot accurately determine the appropriate distance between the two vehicles, resulting in poor control effectiveness. Summary of the Invention

[0004] In view of this, the object of the present invention is to provide a two-vehicle ranging method, ranging device, system and computer-readable storage medium, which can improve the problem that existing target detection methods cannot determine whether the distance between two cooperating vehicles is appropriate, resulting in poor control effect.

[0005] In a first aspect, the present invention provides a two-vehicle distance measurement method, which adopts the following technical solution.

[0006] A two-vehicle ranging method is applied to a ranging device, wherein the ranging device is equipped with a binocular camera and is disposed on a first vehicle, the first vehicle and the second vehicle working in cooperation. The method comprises:

[0007] acquiring in real time a character image of the second vehicle captured by the binocular camera;

[0008] Obtaining a point cloud image of the character image;

[0009] The center coordinates of the point cloud image are calculated, and the distance between the first vehicle and the second vehicle is calculated based on the center coordinates.

[0010] Furthermore, the step of obtaining the point cloud image of the character image includes:

[0011] Using a target detection network model to detect the position of a character in the character image, obtaining coordinate information of the character, and obtaining a character area based on the coordinate information;

[0012] An RGB image and a depth image of the character area are intercepted, and a point cloud image is generated according to the RGB image and the depth image.

[0013] Furthermore, the target detection network model includes a Yolov5 network model, and the step of using the target detection network model to detect the position of the character in the character image, obtaining the coordinate information of the character, and obtaining the character area according to the coordinate information includes:

[0014] Segmenting the character image into a grid comprising S*S cells;

[0015] Predicting, based on each cell, a plurality of bounding boxes of characters whose center points fall within the cell, obtaining a predicted value for each bounding box, the predicted value including coordinate information and a confidence level;

[0016] Based on the confidence level, determining target coordinate information according to the predicted values of all the bounding boxes;

[0017] According to the target coordinate information, angular coordinates are obtained to determine the character area according to the angular coordinates.

[0018] Furthermore, the step of determining target coordinate information based on the confidence level and the predicted values of all the bounding boxes includes:

[0019] Comparing the confidence scores of all the bounding boxes, and determining the bounding box with the highest confidence score as the comparison box;

[0020] Using the intersection-over-union loss function, the intersection-over-union ratios of the comparison box and the remaining bounding boxes are calculated in sequence;

[0021] Comparing each of the intersection-and-union ratios with a preset threshold, and if no intersection-and-union ratio is greater than the preset threshold, taking the coordinate information of the comparison frame as the target coordinate information;

[0022] Otherwise, the comparison frame is removed, and a new comparison frame is determined in the remaining bounding frames until the target coordinate information is obtained.

[0023] Furthermore, the step of obtaining the angular coordinates according to the target coordinate information includes:

[0024] Based on the target coordinate information, a coordinate calculation formula is used to obtain angular coordinates, wherein the angular coordinates include upper left corner coordinates (u0, v0), upper right corner coordinates (u1, v1), lower right corner coordinates (u2, v2) and lower left corner coordinates (u3, v3);

[0025] The coordinate calculation formula includes:

[0026]

[0027] Among them, (x, y, w, h) is the target coordinate information, x represents the horizontal coordinate of the center point of the character area, y represents the vertical coordinate of the center point of the character area, w represents the width of the character area, and h represents the height of the character area.

[0028] Furthermore, the step of generating a point cloud image according to the RGB image and the depth image includes:

[0029] Obtain pixel coordinates according to the RGB image, and obtain depth data according to the depth image;

[0030] Based on the pixel coordinates and the depth data, combined with the binocular camera parameters, a spatial point coordinate calculation formula is used to obtain the spatial point coordinates;

[0031] Generate a point cloud image according to the spatial point coordinates;

[0032] The spatial point coordinate calculation formula includes:

[0033]

[0034] Among them, c x is the aperture center on the horizontal axis of the binocular camera, c y is the aperture center on the vertical axis of the binocular camera, f x is the focal length of the binocular camera on the horizontal axis, f y is the focal length of the binocular camera on the vertical axis, s is the scaling factor of the depth map, (u, v) is the pixel coordinate, and d is the depth data.

[0035] Furthermore, the step of calculating the center coordinates of the point cloud image includes:

[0036] Based on the coordinates of the four spatial points in the character area, the center coordinates of the point cloud image are calculated using a center point calculation formula;

[0037] The center point calculation formula includes:

[0038]

[0039] Among them, (x mid ,y mid ,z mid ) are the coordinates of the center point of the point cloud, (x0, y0, z0), (x1, y1, z1), (x2, y2, z2) and (x3, y3, z3) are the coordinates of the four spatial points in the character area.

[0040] Furthermore, the step of calculating the distance between the first vehicle and the second vehicle based on the center coordinates includes:

[0041] Taking the first vehicle as the coordinate origin, based on the distance between the coordinate origin and the shown center coordinates, the Euclidean distance formula is used to obtain the vehicle distance between the first vehicle and the second vehicle.

[0042] Furthermore, the second vehicle is provided with a plurality of positioning characters on its body, the first vehicle is matched with a plurality of relative parking positions, and the plurality of positioning characters correspond one-to-one to the plurality of relative parking positions. The method further comprises:

[0043] For each of the positioning characters, the character image of the positioning character on the body of the second vehicle is obtained to obtain the distance between the first vehicle and the second vehicle relative to the positioning character. Based on the distance and the preset value, it is determined whether the first vehicle is parked at the relative parking position corresponding to the positioning character.

[0044] Furthermore, the step of determining whether the first vehicle is parked at the relative parking position corresponding to the positioning character based on the vehicle distance and a preset value includes:

[0045] determining whether the vehicle distance is within a preset value, and if so, sending a work instruction to the first vehicle; otherwise, sending a reminder instruction to the first vehicle;

[0046] The work instruction instructs the first vehicle to park at the relative parking position corresponding to the positioning character, and the reminder instruction indicates that the first vehicle is not parked at the relative parking position corresponding to the positioning character.

[0047] Furthermore, the target detection network model includes an R-CNN module or a Faster RCNN model.

[0048] In a second aspect, the present invention provides a distance measuring device, which adopts the following technical solution.

[0049] A ranging device includes a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the method according to the first aspect.

[0050] The third solution of the present invention provides a ranging system, which adopts the following technical solution.

[0051] A ranging system includes a first vehicle and a second vehicle, wherein the first vehicle includes a ranging device, and the ranging device implements the method described in the first aspect.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium, which adopts the following technical solution.

[0053] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method according to the first aspect.

[0054] The two-vehicle ranging method, ranging device, system, and computer-readable storage medium provided in the embodiments of the present invention are based on a three-dimensional character image of the second vehicle captured by a binocular camera, and then obtain a point cloud map of the character image. The distance between the first vehicle and the second vehicle is calculated using the center coordinates of the point cloud map, thereby being able to obtain the distance between the first vehicle and the second vehicle in real time and more accurately, thereby improving the problem that existing target detection methods cannot determine whether the distance between two cooperating vehicles is appropriate, resulting in poor control effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 A block diagram of a harvesting system provided by an embodiment of the present invention is shown.

[0057] Figure 2 Shown Figure 1 Schematic diagram of the structure when the first vehicle and the second vehicle work together.

[0058] Figure 3 A block diagram of a distance measuring device provided by an embodiment of the present invention is shown.

[0059] Figure 4 The figure shows a flow chart of a two-vehicle ranging method provided by an embodiment of the present invention.

[0060] Figure 5 A schematic diagram of a process for cooperating between a first vehicle and a second vehicle to unload grain provided by an embodiment of the present invention is shown.

[0061] Figure 6 Another schematic flow chart of a two-vehicle ranging method according to an embodiment of the present invention is shown.

[0062] Figure 7 Shown Figure 6 Flow chart of some sub-steps of step S103-1.

[0063] Figure 8Shown Figure 7 Flow chart of some sub-steps of step S203.

[0064] Figure 9 Shown Figure 6 Flow chart of some sub-steps of step S103-2.

[0065] Figure 10 A block diagram of a distance measuring device provided by an embodiment of the present invention is shown.

[0066] Icons: 10-harvesting system; 100-first vehicle; 110-second vehicle; 120-ranging device; 130-binocular camera; 140-control device; 150-communication module; 160-processor; 170-memory; 190-ranging device; 200-image acquisition module; 210-detection module; 220-merging module; 230-computing module. DETAILED DESCRIPTION

[0067] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in a variety of different configurations.

[0068] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0069] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0070] Unmanned agricultural production is a key development direction for modern agriculture. From a longer-term perspective, the loss of labor due to an aging population poses a significant challenge to traditional, labor-intensive agriculture. Unmanned agricultural production uses intelligent robots and management systems to help agricultural managers efficiently manage production. Through in-depth insights into all aspects of agricultural production and ongoing exploration of innovative applications of autonomous driving, artificial intelligence, and the Internet of Things in various agricultural scenarios, it empowers agriculture with technology, helping farmers reduce costs and increase efficiency, freeing them up to work.

[0071] Traditional object detection methods are divided into three parts: region selection, feature extraction, and classifier. Deep learning object detection is divided into two categories. One is the two-stage method, which divides the entire process into two parts: generating candidate boxes and identifying objects within the boxes. The other is the one-stage method, which unifies the entire process and directly provides detection results. It mainly includes the Yolo and SSD series.

[0072] In unmanned agricultural production, two or more unmanned vehicles often work in coordination, such as a harvester and a grain transfer vehicle. The harvester harvests rice and, after harvesting a certain amount, unloads the rice onto the grain transfer vehicle. The grain transfer vehicle then transports the rice. When the harvester unloads the rice onto the grain transfer vehicle, strict requirements exist for the distance between the two vehicles: the distance must be within a preset value. If the distance exceeds this value, the rice will leak. Currently, traditional target detection methods or GPS positioning are commonly used to determine the distance between the harvester and the grain transfer vehicle. However, these traditional target detection and GPS positioning methods have poor positioning accuracy and cannot accurately determine whether the distance between the two cooperating vehicles is appropriate, resulting in poor control effectiveness.

[0073] Based on this, the present invention provides a two-vehicle ranging method, ranging device, system and computer-readable storage medium.

[0074] Please refer to Figure 1 and Figure 2 The two-vehicle ranging method provided by the present invention can be applied to a harvesting system 10, which includes a first vehicle 100, a second vehicle 110, a binocular camera 130, and a ranging device 120. The binocular camera 130 is mounted on the ranging device 120, and a counting device is provided on the first vehicle 100. The first vehicle 100 and the second vehicle 110 work in conjunction with each other. A control device 140 is provided on each of the first vehicle 100 and the second vehicle 110. The control device on the first vehicle 100 is in communication with the ranging device 120 and the control device 140 on the second vehicle 110, and the ranging device 120 is in communication with the binocular camera 130. Specifically, the first vehicle 100 can be a harvesting vehicle, and the second vehicle 110 can be a grain transfer vehicle.

[0075] Please refer to Figure 3 is a block diagram of the ranging device 120. The ranging device 120 includes a memory 170, a processor 160, and a communication module 150. The memory 170, processor 160, and communication module 150 are electrically connected to each other, directly or indirectly, to enable data transmission or exchange. For example, these components may be electrically connected via one or more communication buses or signal lines.

[0076] The memory 170 is used to store programs or data. The memory 170 may be, but is not limited to, a random access memory 170 (RAM), a read-only memory 170 (ROM), a programmable read-only memory 170 (PROM), an erasable programmable read-only memory 170 (EPROM), an electrically erasable programmable read-only memory 170 (EEPROM), etc.

[0077] The processor 160 is used to read / write data or programs stored in the memory 170 and execute corresponding functions.

[0078] The communication module 150 is used to establish a communication connection between the ranging device 120 and other communication terminals through a network, and to send and receive data through the network.

[0079] Specifically, the memory 170 stores machine-executable instructions that can be executed by the processor 160 , and the processor 160 can execute the machine-executable instructions in the memory 170 to implement the two-vehicle ranging method provided by the present invention.

[0080] It should be understood that Figure 3 The structure shown is only a schematic diagram of the distance measuring device 120. The distance measuring device 120 may also include a Figure 3 More or fewer components than shown, or with Figure 3 Different configurations shown. Figure 3 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0081] In one embodiment, see Figure 4 , is a two-vehicle distance measurement method provided by the present invention. This embodiment mainly uses this method for Figure 1The distance measuring device 120 in FIG. 1 is used as an example. It is understood that the method can also be used in a server or a terminal. In this embodiment, the method includes the following steps.

[0082] S101, acquiring a character image of a second vehicle captured by a binocular camera in real time.

[0083] Specifically, the control device 140 on the first vehicle 100 sends a ranging instruction to the ranging device 120. The ranging device 120 responds to the ranging instruction and controls the binocular camera 130 to capture the character image of the second vehicle 110 in real time. The ranging device 120 obtains the character image captured by the binocular camera 130 in real time.

[0084] In addition, it is well known that the binocular camera 130 performs three-dimensional scanning to shoot, and the image obtained by shooting is a three-dimensional image. Therefore, the character image is a three-dimensional image, that is, an image of a three-dimensional matrix.

[0085] S103: Obtain a point cloud image of the character image.

[0086] Specifically, the distance measuring device 120 obtains a point cloud image from the character image.

[0087] S105 , calculating the center coordinates of the point cloud image, and calculating the distance between the first vehicle and the second vehicle based on the center coordinates.

[0088] Specifically, the distance measuring device 120 calculates the center coordinates of the point cloud image, and calculates the distance between the first vehicle 100 and the second vehicle 110 based on the center coordinates.

[0089] In the above-mentioned two-vehicle ranging method, a three-dimensional character image of the body of the second vehicle 110 is captured by the binocular camera 130, and then a point cloud map of the character image is obtained. The distance between the first vehicle 100 and the second vehicle 110 is calculated by the center coordinates of the point cloud map, so that the distance between the first vehicle 100 and the second vehicle 110 can be obtained in real time and more accurately, so as to improve the problem that the existing target detection method cannot determine whether the distance between the two cooperating vehicles is appropriate, resulting in poor control effect.

[0090] During the coordinated operation of the grain transfer vehicle and the harvester, the harvester typically needs to move multiple times, switching positions to unload rice onto the grain transfer vehicle, to ensure a uniform loading of the grain transfer vehicle's carrying box. To this end, the second vehicle 110 can be provided with multiple positioning characters, and the first vehicle 100 can be provided with multiple relative docking positions. The multiple positioning characters correspond one-to-one to the multiple relative docking positions, and the multiple positioning characters are evenly spaced along the length of the carrying box.

[0091] On this basis, the two-vehicle distance measurement method may include: for each positioning character, by obtaining the character image of the positioning character on the body of the second vehicle 110, the distance between the first vehicle 100 and the second vehicle 110 relative to the positioning character is obtained, and based on the distance and the preset value, it is determined whether the first vehicle 100 is parked at the relative parking position corresponding to the positioning character.

[0092] Specifically, refer to Figure 5 Assume the number of positioning characters is 4, L is a preset value, and D is the distance between vehicles. When the first vehicle 100 is preparing to unload grain, it approaches the position of the first positioning character of the second vehicle 110. The ranging device 120 begins executing steps S101-S105 to obtain the distance between the first positioning character, that is, the distance between the first vehicle 100 and the second vehicle 110 with respect to the first positioning character. If the distance between the first positioning character and the second vehicle 110 is within the preset value, a work instruction is sent to the first vehicle 100; otherwise, a reminder instruction is sent to the first vehicle 100. Specifically, a reminder instruction is sent if D is within L.

[0093] After receiving the reminder instruction, the first vehicle 100 adjusts its own position and repeats steps S101 to S105 to continue obtaining the vehicle distance of the first positioning character until receiving the work instruction.

[0094] After receiving the work order, the first vehicle 100 starts to unload the grain to the second vehicle 110. After unloading the grain, please continue to refer to Figure 5 First vehicle 100 moves to a position close to the second positioning character on the body of second vehicle 110. Distance measuring device 120 continues to execute steps S101-S105 to obtain the distance between the vehicles of the second positioning character, that is, the distance between first vehicle 100 and second vehicle 110 with respect to the first positioning character. The device determines whether the distance between the vehicles of the first positioning character is within a preset value, and then sends a work instruction to first vehicle 100. Otherwise, it sends a reminder instruction to first vehicle 100.

[0095] The working instruction instructs the first vehicle 100 to park at the relative parking position corresponding to the positioning character, and the re-reminding instruction indicates that the first vehicle 100 has not parked at the relative parking position corresponding to the positioning character.

[0096] This process is repeated until the grain transfer is completed. The sign indicating the completion of grain transfer can be a number of grain unloading operations corresponding to the number of positioning characters, or the completion of grain unloading on the first vehicle 100.

[0097] When the first vehicle 100 is a harvesting vehicle and the second vehicle 110 is a grain transfer vehicle, the preset value may be 1.5 m to 2 m.

[0098] It should be understood that when obtaining the vehicle distance of the first positioning character, the binocular camera 130 captures the character image of the first positioning character, and when obtaining the vehicle distance of the second positioning character, the binocular camera 130 captures the character image of the second positioning character.

[0099] In addition, in this embodiment, the content of the cooperation between the first vehicle 100 and the second vehicle 110 is grain harvesting and grain transportation, which is only an example and not the only limitation. In other embodiments, the content of the cooperation between the first vehicle 100 and the second vehicle 110 can be any other work.

[0100] When the first vehicle 100 is a harvesting vehicle and the second vehicle 110 is a grain transfer vehicle, the distance measuring device 120 equipped with a binocular camera 130 can be set on the grain unloading drum arm of the harvesting vehicle.

[0101] In one embodiment, for S103, refer to Figure 6 , Figure 6 The flowchart including some sub-steps of S103 includes the following steps.

[0102] S103-1, using the target detection network model to detect the position of the character in the character image, obtain the coordinate information of the character, and obtain the character area based on the coordinate information.

[0103] Specifically, the distance measuring device 120 uses the target detection network model to detect the position of the character in the character image, obtains the coordinate information of the character, and obtains the character area according to the coordinate information.

[0104] S103-2, intercepting the RGB image and depth image of the character area, and generating a point cloud image based on the RGB image and the depth image.

[0105] Since the character image itself is a three-dimensional image, that is, a three-dimensional matrix, the character area with the character image is also an image of the three-dimensional matrix. Therefore, the RGB image and the depth image can be obtained based on the character area.

[0106] Specifically, the distance measuring device 120 intercepts the RGB image and the depth image of the character area, and generates a point cloud image according to the RGB image and the depth image.

[0107] After detecting the position of the characters through the target detection network model, a more accurate character area is obtained, and a point cloud map is generated based on the RGB image and depth map of the character Wu Qianyu, which can make the point cloud map more accurate and help improve the accuracy of vehicle distance.

[0108] In one embodiment, the object detection network model in step S103-1 may include a YOLOv5 network model. The core concept of the Yolo network model is to use a convolutional neural network (CNN) network to segment the input image into an S*S grid, i.e., the grid includes S*S cells, and then detect the object whose center point falls within each cell based on each cell.

[0109] Based on the target detection network model including the Yolov5 network model, refer to Figure 7 , Figure 7 This is a flowchart of some sub-steps of step S103-1, including the following sub-steps.

[0110] S201, dividing the character image into a grid consisting of S*S cells.

[0111] S202 , based on each cell, predict multiple bounding boxes of characters whose center points fall within the cell, and obtain a predicted value for each bounding box.

[0112] The predicted value includes coordinate information and confidence level.

[0113] Specifically, each bounding box can be represented by four values: (x, y, w, h), where (x, y) represents the center coordinates of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box. (x, y, w, h) represents the coordinate information. The confidence level is represented by c. In practice, the predicted value for each bounding box consists of five elements: (x, y, w, h, c).

[0114] On this basis, if the character image is divided into an S*S grid and each cell predicts (B*5+C) values, then the final predicted value is a tensor of S*S*(B*5+C), that is, as an array.

[0115] S203: Based on the confidence level, target coordinate information is determined according to the predicted values of all bounding boxes.

[0116] The target coordinate information is the center coordinate of the character area to be obtained.

[0117] S204: Obtain angular coordinates according to the target coordinate information, and determine the character area according to the angular coordinates.

[0118] The angular coordinates include the four corner coordinates of the character area. Therefore, after obtaining the angular coordinates, the position of the character area can be determined.

[0119] It can be understood that the character area is an area containing characters.

[0120] The Yolov5 network model has the advantages of less detection time and higher accuracy. The Yolov5 network model can obtain the bounding box more quickly and accurately, thereby improving the acquisition speed and accuracy of the character area.

[0121] Further, refer to Figure 8 , Figure 8 This is a flowchart of some sub-steps of the above step S203, including the following steps.

[0122] S203-1: Compare the confidences of all bounding boxes and determine the bounding box with the highest confidence as the comparison box.

[0123] S203-2, using the intersection-over-union loss function, sequentially calculate the intersection-over-union ratios of the comparison box and the remaining bounding boxes.

[0124] S203-3, comparing each intersection-to-union ratio with a preset threshold.

[0125] If no intersection-over-union ratio is greater than the preset threshold, step S203 - 4 is executed; otherwise, step S203 - 5 is executed.

[0126] S203-4: Use the coordinate information of the comparison frame as the target coordinate information.

[0127] S203-5, remove the comparison box, determine a new comparison box in the remaining bounding boxes, and return to step S203-1 until the target coordinate information is obtained.

[0128] If the IoU value is greater than the preset threshold, it means that the comparison box has a high degree of overlap with the bounding box from which the IoU value was derived. Therefore, removing the comparison box can reduce the overlap. If no IoU value is greater than the preset threshold, it indicates that no bounding box has a high degree of overlap with the comparison box. Therefore, the coordinates of the comparison box with the highest confidence level can be used as the target coordinates.

[0129] The Yolo algorithm regards target detection as a regression model and uses a mean square error loss function, which makes the position of the bounding box more accurate, thereby helping to improve the accuracy of the character area.

[0130] In the algorithm of the Yolov5 network model, the confidence calculation formula is:

[0131] in, represents the confidence of the j-th bounding box of the i-th cell. Refers to the intersection-and-union ratio of the predicted current bounding box and the object's actual bounding box when the current bounding box contains an object. Pr(Object) represents the probability of whether the current bounding box contains an object. The methods for obtaining Pr(Object) and Pr(Object) are common algorithms in Yolov5 and will not be described in detail here.

[0132] In other embodiments, the selectable target detection network model also includes detection models such as R-CNN or Faster RCNN, which can also perform target detection and obtain character regions.

[0133] In other embodiments, the original point cloud image of the characters on the second vehicle 110 can be directly read, and based on the original point cloud image, a three-dimensional target detection model is introduced to identify the character coordinates of the body of the second vehicle 110, and the identified character coordinates are substituted into the Euclidean distance formula to calculate the spatial distance between the first vehicle 100 and the second vehicle 110, that is, the vehicle distance.

[0134] Since the target coordinate information is the coordinate information of the comparison frame, the target coordinate information is also (x, y, w, h). On this basis, in step S103-4, the method for obtaining the angular coordinates according to the target coordinate information can be:

[0135] Based on the target coordinate information, the coordinate calculation formula is used to obtain the angular coordinates.

[0136] The corner coordinates include the upper left corner coordinates (u0, v0), the upper right corner coordinates (u1, v1), the lower right corner coordinates (u2, v2), and the lower left corner coordinates (u3, v3). Furthermore, since the target coordinate information is the coordinate information of a comparison frame, the target coordinate information at this time is also (x, y, w, h).

[0137] The coordinate calculation formula includes:

[0138]

[0139] In the above coordinate calculation formula, x represents the horizontal coordinate of the center point of the character area, y represents the vertical coordinate of the center point of the character area, w represents the width of the character area, and h represents the height of the character area.

[0140] That is, the obtained target coordinate information is actually the coordinates of the center point of the character area plus the length and width of the character area.

[0141] According to the above coordinate calculation formula, after obtaining the target coordinate information, the four corner coordinates can be quickly calculated. The four corner coordinates are the four corner coordinates of the character area, so the character area can be determined.

[0142] In one embodiment, reference Figure 9 , Figure 9 This is a flowchart of some sub-steps of step S103 - 2 , through which a point cloud image is generated according to the RGB image and the depth image.

[0143] S301, obtaining pixel coordinates according to the RGB image, and obtaining depth data according to the depth image.

[0144] After obtaining the RGB image, the pixel coordinates can be directly read out, and after obtaining the depth map, the depth data can be directly obtained.

[0145] S302 , based on the pixel coordinates and depth data, combined with the binocular camera parameters, a spatial point coordinate calculation formula is used to obtain the spatial point coordinates.

[0146] The spatial coordinate calculation formula includes:

[0147]

[0148] Among them, c x is the aperture center on the horizontal axis of the binocular camera 130, c y is the aperture center on the vertical axis of the binocular camera 130, f x is the focal length of the binocular camera 130 on the horizontal axis, f y is the focal length of the binocular camera 130 on the vertical axis, s is the scaling factor of the depth map, (u, v) is the pixel coordinate, and d is the depth data.

[0149] Specifically, substituting the pixel coordinates and depth data of the four corners of the character area into the spatial coordinate calculation formula yields the four spatial coordinates of the point cloud: (x0, y0, z0), (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3). The pixel coordinates of the four corners of the character area are the top-left corner (u0, v0), top-right corner (u1, v1), bottom-right corner (u2, v2), and bottom-left corner (u3, v3).

[0150] S303: Generate a point cloud image according to the spatial point coordinates.

[0151] After obtaining pixel coordinates and depth data through the RGB image and the depth image, the spatial coordinates can be quickly obtained by combining the binocular camera 130 parameters and the spatial coordinate calculation formula, and the point cloud image can be determined based on the spatial coordinates.

[0152] The generated point cloud image is a three-dimensional image of the body characters of the second vehicle 110. The three-dimensional spatial coordinates of the grain transfer vehicle can be known by calculating the coordinates of the center point of the point cloud image.

[0153] On the basis of the above, the method for calculating the center coordinates of the point cloud in step S107 includes: calculating the center coordinates of the point cloud using a center point calculation formula based on the coordinates of the four spatial points in the character area.

[0154] The center point calculation formula includes:

[0155]

[0156] Among them, (x mid ,y mid ,z mid ) are the coordinates of the center point of the point cloud, (x0, y0, z0), (x1, y1, z1), (x2, y2, z2) and (x3, y3, z3) are the coordinates of the four spatial points in the character area.

[0157] Then, after the center coordinates of the point cloud are obtained, step S105 of calculating the distance between the first vehicle 100 and the second vehicle 110 according to the center coordinates includes:

[0158] Taking the first vehicle 100 as the coordinate origin, based on the distance between the coordinate origin and the center coordinate, the Euclidean distance formula is used to obtain the vehicle distance between the first vehicle 100 and the second vehicle 110.

[0159] Specifically, the distance between the first vehicle 100 and the second vehicle 110 is calculated using the position of the binocular camera 130 on the first vehicle 100 as the coordinate origin.

[0160] Among them, the three-dimensional space Euclidean distance formula is: Since the first vehicle 100 is the coordinate origin, the first vehicle 100 where the binocular camera 130 is located is used as a reference. Therefore, the vehicle distance is:

[0161] Furthermore, the control devices 140 of the first vehicle 100 and the second vehicle 110 may be provided with display interfaces for displaying the working instructions and reminder instructions sent by the ranging device 120 .

[0162] The two-vehicle ranging method provided in this embodiment detects a positioning character using a target detection network model such as the Yolov5 network model, obtains a character region based on the positioning character, generates a point cloud image based on the RGB image and depth image of the character region, calculates the center coordinates of the point cloud image (i.e., the center point coordinates), and calculates the distance from the center point coordinates to the first vehicle 100, i.e., the distance between the two vehicles, using the Euclidean distance formula. The judgment result (operation instruction or reminder instruction) between the distance and a preset value is then returned to the control device 140 of the first vehicle 100 to control the operation of the first vehicle 100 and the second vehicle 110. This helps to control the distance between the first vehicle 100 and the second vehicle 110 within an appropriate range when the first vehicle 100 and the second vehicle 110 are working together, thereby improving the control effect.

[0163] It should be understood that although Figure 4-Figure 9The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 4-Figure 9 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0164] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing the distance measuring device 190 is given below. Optionally, the distance measuring device 190 can adopt the above Figure 3 The device structure of the distance measuring device 120 is shown. Figure 10 , Figure 10 This is a functional block diagram of a distance measuring device 190 provided in an embodiment of the present invention. It should be noted that the basic principles and technical effects of the distance measuring device 190 provided in this embodiment are the same as those of the aforementioned two-vehicle distance measuring method. For the sake of brevity, any details not mentioned in this embodiment are referred to the corresponding content in the aforementioned embodiments. The distance measuring device 190 includes an image acquisition module 200, a detection module 210, a merging module 220, and a calculation module 230. The distance measuring device 190 may also include a binocular camera 130 and may be in communication with the binocular camera 130.

[0165] The image acquisition module 200 is used to obtain in real time the character image of the body of the second vehicle 110 captured by the binocular camera 130.

[0166] The detection module 210 is used to detect the position of the characters in the character image using the target detection network model, obtain the coordinate information of the characters, and obtain the character area according to the coordinate information.

[0167] The merging module 220 is used to intercept the RGB image and the depth image of the character area and generate a point cloud image based on the RGB image and the depth image.

[0168] The calculation module 230 is used to calculate the center coordinates of the point cloud image and calculate the distance between the first vehicle 100 and the second vehicle 110 based on the center coordinates.

[0169] Optionally, the above modules can be stored in the form of software or firmware. Figure 3The memory 170 shown in FIG. 1 is stored in the operating system (OS) of the distance measuring device 120 and can be used by Figure 3 Meanwhile, the data, program codes, etc. required to execute the above modules may be stored in the memory 170.

[0170] In one embodiment, the present invention further provides a ranging system, which includes a first vehicle 100 and a second vehicle 110. The first vehicle 100 includes a ranging device 120. The ranging device implements the two-vehicle ranging method provided by the present invention to measure the distance between the first vehicle 100 and the second vehicle 110.

[0171] Specifically, the distance measuring device has the same functional modules as the distance measuring apparatus 190 described above.

[0172] Furthermore, the ranging device 120 may be installed on the body of the first vehicle 110 .

[0173] In the above-mentioned ranging system, the ranging device 120 on the first vehicle 100 obtains a point cloud map of the character image based on the three-dimensional character image of the second vehicle 110 captured by the binocular camera 130, and calculates the distance between the first vehicle 100 and the second vehicle 110 using the center coordinates of the point cloud map, thereby being able to obtain the distance between the first vehicle 100 and the second vehicle 110 in real time and more accurately. In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the apparatuses, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

[0174] In addition, the functional modules in various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0175] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0176] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A two-vehicle ranging method, characterized in that: Applied to a distance measuring device, the distance measuring device is equipped with a binocular camera, the distance measuring device is set on a first vehicle, and the first vehicle works in conjunction with a second vehicle. The method includes: acquiring in real time a character image of the second vehicle captured by the binocular camera; Obtaining a point cloud image of the character image; Calculating the center coordinates of the point cloud image, and calculating the distance between the first vehicle and the second vehicle based on the center coordinates; The second vehicle is provided with a plurality of positioning characters on its body, the first vehicle is matched with a plurality of relative parking positions, the plurality of positioning characters correspond one-to-one to the plurality of relative parking positions, and the method further comprises: For each positioning character, obtaining an image of the positioning character on the body of the second vehicle to obtain a distance between the first vehicle and the second vehicle relative to the positioning character, and determining whether the first vehicle is parked at the relative parking position corresponding to the positioning character based on the distance and a preset value; The step of determining whether the first vehicle is parked at the relative parking position corresponding to the positioning character based on the vehicle distance and a preset value includes: determining whether the vehicle distance is within a preset value, and if so, sending a work instruction to the first vehicle; otherwise, sending a reminder instruction to the first vehicle; The work instruction instructs the first vehicle to park at the relative parking position corresponding to the positioning character, and the reminder instruction indicates that the first vehicle is not parked at the relative parking position corresponding to the positioning character.

2. The two-vehicle distance measurement method according to claim 1, characterized in that: The step of obtaining a point cloud image of the character image includes: Using a target detection network model to detect the position of a character in the character image, obtaining coordinate information of the character, and obtaining a character area based on the coordinate information; An RGB image and a depth image of the character area are intercepted, and a point cloud image is generated according to the RGB image and the depth image.

3. The two-vehicle distance measurement method according to claim 2, characterized in that: The target detection network model includes a Yolov5 network model. The target detection network model is used to detect the position of the character in the character image, obtain the coordinate information of the character, and obtain the character area according to the coordinate information, including: Segmenting the character image into a grid comprising S*S cells; Predicting, based on each cell, a plurality of bounding boxes of characters whose center points fall within the cell, obtaining a predicted value for each bounding box, the predicted value including coordinate information and a confidence level; Based on the confidence level, determining target coordinate information according to the predicted values of all the bounding boxes; According to the target coordinate information, angular coordinates are obtained to determine the character area according to the angular coordinates.

4. The two-vehicle distance measurement method according to claim 3, characterized in that: The step of determining target coordinate information based on the confidence level and the predicted values of all the bounding boxes includes: Comparing the confidence scores of all the bounding boxes, and determining the bounding box with the highest confidence score as the comparison box; Using the intersection-over-union loss function, the intersection-over-union ratios of the comparison box and the remaining bounding boxes are calculated in sequence; Comparing each of the intersection-and-union ratios with a preset threshold, and if no intersection-and-union ratio is greater than the preset threshold, taking the coordinate information of the comparison frame as the target coordinate information; Otherwise, the comparison frame is removed, and a new comparison frame is determined in the remaining bounding frames until the target coordinate information is obtained.

5. The two-vehicle distance measurement method according to claim 3, characterized in that: The step of obtaining angular coordinates according to the target coordinate information includes: Based on the target coordinate information, a coordinate calculation formula is used to obtain angular coordinates, wherein the angular coordinates include upper left corner coordinates (u0, v0), upper right corner coordinates (u1, v1), lower right corner coordinates (u2, v2) and lower left corner coordinates (u3, v3); The coordinate calculation formula includes: Among them, (x, y, w, h) is the target coordinate information, x represents the horizontal coordinate of the center point of the character area, y represents the vertical coordinate of the center point of the character area, w represents the width of the character area, and h represents the height of the character area.

6. The two-vehicle distance measurement method according to claim 2, characterized in that: The step of generating a point cloud image according to the RGB image and the depth image includes: Obtain pixel coordinates according to the RGB image, and obtain depth data according to the depth image; Based on the pixel coordinates and the depth data, combined with the binocular camera parameters, a spatial point coordinate calculation formula is used to obtain the spatial point coordinates; Generate a point cloud image according to the spatial point coordinates; The spatial point coordinate calculation formula includes: Among them, c x is the aperture center on the horizontal axis of the binocular camera, c y is the aperture center on the vertical axis of the binocular camera, f x is the focal length of the binocular camera on the horizontal axis, f y is the focal length of the binocular camera on the vertical axis, s is the scaling factor of the depth map, (u, v) is the pixel coordinate, and d is the depth data.

7. The two-vehicle distance measurement method according to claim 1, characterized in that: The step of calculating the center coordinates of the point cloud image includes: Based on the coordinates of the four spatial points in the character area, the center coordinates of the point cloud image are calculated using a center point calculation formula; The center point calculation formula includes: Among them, (x mid ,y mid ,z mid ) are the coordinates of the center point of the point cloud, (x0, y0, z0), (x1, y1, z1), (x2, y2, z2) and (x3, y3, z3) are the coordinates of the four spatial points in the character area.

8. The two-vehicle distance measurement method according to claim 1, characterized in that: The step of calculating the distance between the first vehicle and the second vehicle based on the center coordinates includes: Taking the first vehicle as the coordinate origin, based on the distance between the coordinate origin and the shown center coordinates, the Euclidean distance formula is used to obtain the vehicle distance between the first vehicle and the second vehicle.

9. The two-vehicle distance measurement method according to claim 2, characterized in that: The target detection network model includes an R-CNN module or a Faster RCNN model.

10. A distance measuring device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the method according to any one of claims 1 to 9.

11. A ranging system, characterized in that: The method comprises a first vehicle and a second vehicle, wherein the first vehicle comprises a distance measuring device, and the distance measuring device implements the method according to any one of claims 1 to 9.

12. The system according to claim 11, wherein: The distance measuring device is installed on the body of the first vehicle.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images