Method for matching multiple target images, environmental target recognition method and mine unmanned vehicle

Abstract

The present disclosure provides a method for matching a plurality of target images of a plurality of sensed targets, an identification method of environmental targets, and a mine area unmanned vehicle. The method matches target images of adjacent time points by matching the matching relationship of the azimuth angle for imaging of the target images of adjacent time points.

Description

Technical Field

[0001] This invention relates to the field of mining, specifically to image matching methods, target recognition methods, and unmanned vehicles in mining areas. Background Technology

[0002] With the country's vigorous promotion of intelligent mining construction, 5G-based smart mine unmanned driving has developed rapidly. The deployment and operation of unmanned vehicles in mining areas is the future development trend of open-pit mining scenarios, and the ability to perceive road conditions is particularly important for the actual deployment of unmanned vehicles in mining areas.

[0003] Open-pit mines are dusty and have complex road conditions, which presents new challenges to the environmental perception of unmanned vehicles. Summary of the Invention

[0004] One aspect of the present invention provides a method for matching multiple target images of multiple sensed targets, comprising:

[0005] Acquire a first image, the first image comprising multiple first target images of multiple targets; and

[0006] Based on the matching relationship between the azimuth angle of the first target image at the current moment and the azimuth angle of the first target image at the previous moment for each of the plurality of first target images, a first target image at the current moment that matches the first target image at the previous moment for each of the plurality of targets is determined; and / or based on the distance difference between the positions of the target center points of each of the plurality of targets at the previous moment and the current moment, a first target image at the current moment that matches the first target image at the previous moment for each of the plurality of targets is determined.

[0007] The azimuth angle of each of the plurality of first target images in the spatial coordinate system including the plurality of targets is calculated based on the first imaging parameters that form the first image and the position of the first target image in the first image.

[0008] In one embodiment, calculating the azimuth angle of each of the plurality of first target images based on the first imaging parameters and the position of the first target image in the first image includes:

[0009] Based on the position of each of the plurality of first target images in the first image and the imaging focal length, calculate the azimuth angle of the first target image corresponding to each of the plurality of first target images;

[0010] The matching relationship between the azimuth angle of the first target image at the current moment and the azimuth angle of the first target image at the previous moment is based on the change in the difference between the azimuth angle of the first target image at the current moment and the azimuth angle at the previous moment.

[0011] In one embodiment, the method further includes:

[0012] Acquire at least one other image, the at least one other image comprising multiple at least one other target images of multiple targets, and calculate the azimuth angle of each of the multiple at least one other target images based on another imaging parameter forming the at least one other image and the position of the at least one other target image in the at least one other image;

[0013] Wherein, at least one other target image azimuth angle is a target image azimuth angle that has been transformed into the same spatial coordinate system as the first target image azimuth angle.

[0014] In one embodiment, the method further includes: wherein at least one other image comprises at least two images obtained based on signals of at least two different physical properties.

[0015] In one embodiment, calculating the azimuth angle of each of the plurality of at least other target images based on another imaging parameter forming at least another image and the position of at least another target image in at least another image includes:

[0016] Based on the matching relationship between the azimuth angle of the at least other target image at the current moment of each of the plurality of at least other target images and the azimuth angles of the plurality of first target images and / or the azimuth angles of the plurality of at least other target images at the previous moment, and / or based on the distance difference between the positions of the center points of each of the plurality of targets at the current moment and the previous moment, the at least other target image at the current moment that matches the at least other target image at the previous moment of each of the plurality of targets is determined, thereby obtaining the matched first target image information and at least other target image information of each of the plurality of targets at the current moment.

[0017] In one embodiment, determining a first target image at a current time that matches a first target image at a previous time for each of the plurality of targets, and determining at least another target image at a current time that matches at least another target image at a previous time for each of the plurality of targets, includes:

[0018] Using the cost matrix, the Hungarian algorithm is employed to perform optimal matching to determine the first target image at the current time and at least one other target image at the current time.

[0019] The difference between the azimuth angle of the first target image at the current time and the previous time, and / or the distance difference between the center points of each of the plurality of targets at the current time and the previous time, and / or the sum of both, are used as parameters of the cost matrix.

[0020] In one embodiment, at least one other target image includes a second target image, which is obtained based on a laser signal; and / or

[0021] At least one other target image, including a third target image, was obtained based on millimeter-wave signals.

[0022] One aspect of the present invention provides a method for identifying environmental targets, comprising:

[0023] A first signal is sensed by a first device to acquire a first image, the first image including multiple first target images of multiple targets;

[0024] Based on the matching relationship between the azimuth angle of the first target image at the current moment and the azimuth angle of the first target image at the corresponding previous moment for each of the plurality of first target images, and / or based on the distance difference between the positions of the target center points of each of the plurality of targets at the previous moment and the current moment, determine the first target image at the current moment that matches the first target image at the previous moment for each of the plurality of targets; and

[0025] First target image sequence information with respect to time is obtained for each of the plurality of targets in order to identify each of the plurality of targets;

[0026] The azimuth angle of each of the plurality of first target images in the spatial coordinate system including the plurality of targets is calculated based on the first imaging parameters that form the first image and the position of the first target image in the first image.

[0027] In one embodiment, calculating the azimuth angle of each of the plurality of first target images based on the first imaging parameters and the position of the first target image in the first image includes:

[0028] Based on the position of each of the plurality of first target images in the first image and the imaging focal length, calculate the azimuth angle of the first target image corresponding to each of the plurality of first target images;

[0029] Among them, the matching relationship between the azimuth angle of the first target image at the current moment and the azimuth angle of the first target image at the previous moment is based on the changing pattern of the difference between the azimuth angle of the first target image at the current moment and the azimuth angle at the previous moment.

[0030] In one embodiment, the method further includes:

[0031] Acquire at least one other image, said at least one other target image comprising multiple targets, and calculate the azimuth angle of each of the multiple targets in said at least one other target image based on another imaging parameter forming the at least one other image and the position of the at least one other target image in said at least one other target image; and

[0032] Based on the matching relationship between the azimuth angle of the at least other target image at the current time of each of the plurality of at least other target images and the azimuth angles of the plurality of first target images and / or the azimuth angles of the plurality of at least other target images at the previous time, and / or based on the distance difference between the positions of the center points of each of the plurality of targets at the current time and the previous time, the at least other target image at the current time that matches the at least other target image at the previous time of each of the plurality of targets is determined, thereby obtaining the matched first target image information and at least other target image information at the current time of each of the plurality of targets;

[0033] Wherein, at least one other target image azimuth angle is a target image azimuth angle that has been transformed into the same spatial coordinate system as the first target image azimuth angle.

[0034] In one embodiment, determining a first target image at a current time that matches a first target image at a previous time for each of the plurality of targets, and determining at least another target image at a current time that matches at least another target image at a previous time for each of the plurality of targets, includes:

[0035] Using the cost matrix, the Hungarian algorithm is employed to perform optimal matching to determine the first target image at the current time and at least one other target image at the current time.

[0036] The difference between the azimuth angle of the first target image at the current time and the previous time, and / or the distance difference between the center points of each of the plurality of targets at the current time and the previous time, and / or the sum of both, are used as parameters of the cost matrix.

[0037] In one embodiment, at least one other target image includes a second target image, which is obtained based on a laser signal; and / or

[0038] At least one other target image, including a third target image, was obtained based on millimeter-wave signals.

[0039] One aspect of the present invention provides an unmanned vehicle for mining areas, comprising:

[0040] A first imaging device is configured to sense a first signal in order to acquire a first image, the first image comprising a plurality of first target images of a plurality of targets, wherein a first target image azimuth angle corresponding to each of the plurality of first target images in a spatial coordinate system including the plurality of targets is calculated based on first imaging parameters forming the first image and the position of the first target image in the first image;

[0041] At least another imaging device, configured to sense at least another signal to acquire at least another image, said at least another image comprising a plurality of at least another target images of a plurality of targets, wherein an azimuth angle of at least another target image corresponding to each of the plurality of at least another target images is calculated based on another imaging parameter forming the at least another image; and

[0042] Controller, the controller is configured to:

[0043] Based on the matching relationship between the azimuth angle of the first target image at the current moment of each of the plurality of first target images and the azimuth angle of the first target image at the corresponding previous moment of each of the plurality of first target images, and / or based on the distance difference between the positions of the target center points of each of the plurality of targets at the previous moment and the current moment, determine the first target image at the current moment that matches the first target image at the previous moment of each of the plurality of targets.

[0044] Based on the matching relationship between the azimuth angle of the at least other target image at the current moment and the azimuth angles of the multiple first target images and / or the azimuth angles of the multiple at least other target images at the previous moment for each of the plurality of targets, and / or based on the distance difference between the positions of the center points of each of the plurality of targets at the current moment and the previous moment, at least another target image at the current moment that matches the at least other target image at the previous moment for each of the plurality of targets is determined, thereby obtaining the matched first target image information and at least another target image information at the current moment for each of the plurality of targets; and

[0045] First target image sequence information with respect to time is obtained for each of the plurality of targets in order to identify each of the plurality of targets;

[0046] The controller is further configured to enable autonomous driving by recognizing the multiple targets. Attached Figure Description

[0047] Figure 1 This is a schematic block diagram illustrating the process of a method according to an embodiment of the present disclosure;

[0048] Figure 2 This is a schematic diagram of the camera imaging optical path according to an embodiment of the present disclosure;

[0049] Figure 3 This is a schematic diagram of the coordinates of point B in the imaging two-dimensional coordinate system according to an embodiment of the present disclosure;

[0050] Figure 4 This is a schematic diagram of the arrangement according to an embodiment of the present disclosure;

[0051] Figure 5 This is a schematic diagram of the optimal matching process of a target image according to an embodiment of the present disclosure;

[0052] Figure 6 This is a schematic diagram illustrating the process of calculating the probability of an unmatched target image according to an embodiment of the present disclosure. Detailed Implementation

[0053] To more clearly illustrate the purpose, technical solutions, and advantages of this disclosure, the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the following description of the embodiments is intended to explain and illustrate the overall concept of this disclosure and should not be construed as limiting the disclosure. In the specification and drawings, the same or similar reference numerals refer to the same or similar parts or components. For clarity, the drawings are not necessarily drawn to scale, and some well-known parts and structures may be omitted from the drawings.

[0054] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The word “a” or “an” does not exclude multiple components. Terms such as “including” or “comprising” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” “right,” “top,” or “bottom,” etc., are used only to indicate relative positional relationships, which may change accordingly when the absolute position of the described object changes. When an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element, or there may be intermediate elements present.

[0055] One embodiment of this disclosure provides a method for matching multiple target images of multiple sensed targets. Figure 1 The process of one embodiment of this disclosure is shown and can be referred to Figure 1Understand the general process of the method; however, it should be known that not every step is necessary.

[0056] In one embodiment, the target can be, for example, a person, obstacle, vehicle, or other object within a mining area. A single frame of optical (first) image of a road or other location within the mining area is acquired using, for example, a camera. This frame may include images of one or more of the aforementioned targets. For instance, the camera captures a (first) image in a single frame that includes a person and a vehicle. The camera can continuously capture images, which can then be combined to form a video. To track the position and speed of each target (person, vehicle, obstacle, etc.) using the captured images or video, information about each target needs to be continuously acquired from these successive sets of images.

[0057] In one embodiment, in the image captured by the camera, the (first) image of the person at the current moment is captured at an azimuth angle θ1, which is the spatial angle of the coordinate position of the person's image in the image captured by the camera relative to the focal point. When, for example, a vehicle moves, the camera mounted on the vehicle moves, and the camera will capture an image at the next moment at an azimuth angle θ1'. As time changes, each moment will correspond to an azimuth angle.

[0058] In other words, the azimuth angle changes continuously with time and is a function of time. The azimuth angle changes with respect to time in a continuous curve. This invention determines the image information of a target, such as a person, at each moment by matching the azimuth angles of adjacent moments, thereby deriving the changes in the person's position and velocity, and other parameters over time, thus achieving the tracking of the target, such as a person.

[0059] Here, the image at each moment can be a frame, the image at the previous moment is a frame at the previous moment, and the image at the current moment is a frame at the current moment.

[0060] Figure 2 This diagram illustrates the calculation of azimuth angle. The image captured by the vehicle's camera comprises multiple pixels, which together form an image of a target such as a person or obstacle. Each pixel has a size p (μm), and the camera has an effective focal length f (mm). Figure 2 In this diagram, A is the center point of a target, such as a person; DF represents the optical axis; O is the center point of a convex lens; point B (pixel) is the image of A (for ease of calculation, a person is represented as point A, or it can be considered as a part of a person, A); and the length of FB along the x-axis of the image coordinate system is expressed in pixels n. x The measurement, in terms of length along the y-axis, is expressed in pixels n. y The angle between directions is denoted by θ. Then the following relationship is satisfied:

[0061]

[0062] The calculation yields:

[0063]

[0064] Reference Figure 3 The coordinates of point B in the two-dimensional coordinate system of the image are calculated as follows:

[0065] x = (-1) j ·n x ·p

[0066] y = (-1) k ·n y ·p

[0067] Wherein: when point B (center) is in the first quadrant of the camera coordinate system, j = 0, k = 0; when point B is in the second quadrant of the camera coordinate system, j = 1, k = 0; when point B is in the third quadrant of the camera coordinate system, j = 1, k = 1; when point B is in the fourth quadrant of the camera coordinate system, j = 0, k = 1.

[0068] Here, the position of point B in the camera coordinate system is denoted as (X). c ,Y c Z c According to the imaging principle of a camera, the following relationship is satisfied:

[0069] x·Z c =f·X c

[0070] y·Z c =f·Y c

[0071] The direction vector of point B can be obtained as follows:

[0072]

[0073] Where (x, y) are the coordinates in the two-dimensional plane of the camera coordinate system, which can be calculated based on parameters such as focal length and number of pixels.

[0074] Based on the positional relationship between the camera coordinate system and the world coordinate system (vehicle coordinate system), corresponding rotations and translations are performed to obtain the unit direction vector in the vehicle coordinate system. The specific calculation process is as follows:

[0075] Let the coordinates of the camera coordinate system containing the direction vector be denoted as (X). c ,Y c Z c ) T The vehicle coordinate system coordinates are (X) w ,Y w Z w ) TThe rotation matrix along the x-axis from the camera coordinate system to the vehicle coordinate system is denoted as R. x The rotation matrix along the y-axis is denoted as R. y The rotation matrix along the z-axis is denoted as R. z The translation vector from the camera coordinate system to the world coordinate system is T.

[0076] The calculation formula is as follows:

[0077]

[0078]

[0079] Adjustments can be written in the following form:

[0080]

[0081] Therefore, based on imaging parameters such as focal length, the position of point B in the image, and pixel size, the azimuth angle of the image captured by the camera at each moment, for example, a person or a part of a person, point A, can be calculated. Each frame (image) includes the position information and azimuth angle information of the target image.

[0082] The images captured by the camera include images of multiple targets, such as people and obstacles. These images should correspond one-to-one with the images of multiple targets in the previous moment's camera image, thus linking the image information of each target (person, obstacle) at different times with the corresponding target to form the target's image information over time. This invention considers that the azimuth angles of these targets' images are different; for example, the imaging azimuth angle of a person is θ1, that of an obstacle is θ2, that of a vehicle is θ3, and that of other targets is θn, etc., and that the variation patterns of each corresponding azimuth angle θ are also different. Based on the existing matching relationship of each target's azimuth angle at adjacent moments, the image information at the current moment can be matched with the image information at the previous moment, thereby obtaining continuous image information of each target (e.g., a person) relative to time. The image information includes the target's position, velocity, and other information. Based on the relationship between the obtained image information and time, tracking of targets such as people can be achieved.

[0083] The azimuth matching relationship reflects the changing pattern of the difference in azimuth angles between the current moment and the previous moment (or adjacent moments), such as the image of a human target. The azimuth matching relationship can be determined, for example, through minimum matching.

[0084] In particular, under conditions of low target density and / or numerous environmental interference factors, such as in mining environments, practice has shown that azimuth-based matching is a very effective method for target image matching.

[0085] Compared to other existing technologies, this method has higher accuracy of azimuth angles when using the same matching algorithm, thus reducing the requirements for the target image matching algorithm and consequently reducing the computational power requirements, thereby greatly saving costs and improving the real-time response.

[0086] In embodiments of the present invention, the vehicle is also equipped with a lidar and / or a millimeter-wave imaging device, which acquire second and third images (relative to the first image captured by the camera), respectively. Similar to the first image captured by the camera, the second and third images include multiple target images of multiple targets. During transportation in mining areas, there is a large amount of dust, and visibility can sometimes be poor, resulting in blurred images captured by the camera. In embodiments of the present invention, by utilizing lidar and / or a millimeter-wave imaging device, more image information can be acquired, improving the speed and efficiency of interference elimination and increasing the ability to identify targets.

[0087] For simplicity, the imaging process of lidar and millimeter-wave imaging devices is similar to the optical path diagram of a camera, but the imaging signals are laser and millimeter waves.

[0088] Similar to a camera, embodiments of the present invention calculate the imaging azimuth angle of the lidar for point A based on parameters such as the focal length of the lidar and the position of point A in the lidar's imaging coordinate system. It should be noted that, according to embodiments of the present invention, the azimuth angle of the lidar for point A (for example, it can be represented by θ2) needs to be converted to a coordinate system that is the same as the imaging coordinate system of the camera, for example, both are converted to the azimuth angle in the vehicle coordinate system (or the Earth coordinate system).

[0089] The same operation applies to millimeter-wave imaging devices, for example... Figure 2 Point A in the model is used to calculate the imaging azimuth angle (e.g., θ3) of the millimeter-wave imaging device when imaging point A, based on parameters such as the position of point A within the imaging coordinate system of the millimeter-wave imaging device. It should be noted that, according to embodiments of the present invention, the azimuth angle of point A of the millimeter-wave imaging device needs to be converted to a coordinate system identical to the imaging coordinate system of the camera, for example, converted to the azimuth angle in the vehicle coordinate system (or the Earth coordinate system).

[0090] According to an embodiment of the present invention, the imaging of the lidar at the current moment is matched with the imaging of the lidar at the previous moment based on the matching relationship between the azimuth angle of the lidar image at the current moment and the previous moment.

[0091] According to an embodiment of the present invention, based on the matching relationship between the azimuth angles of the image formed by the millimeter-wave imaging device at the current moment and the previous moment, the image of the millimeter-wave imaging device at the current moment is matched with the image of the millimeter-wave imaging device at the previous moment.

[0092] The calculation of the azimuth angle for lidar and millimeter-wave imaging devices is basically the same as that for cameras, so it will not be repeated here.

[0093] The imaging azimuth angle of the image formed by the lidar and millimeter-wave imaging device at the previous moment can be the imaging azimuth angle of the image of the target (e.g., a person) formed by the camera at the previous moment; or, the imaging azimuth angle of the image formed by the camera at the previous moment is the same as the imaging azimuth angle of the image formed by the lidar and millimeter-wave imaging device at the previous moment (here it is considered to be the same, although the camera, lidar, and millimeter-wave imaging device are not completely overlapping in space, but the error is ignored). Therefore, according to the embodiments of the present invention, when referring to the same so-called "azimuth angle" at the previous moment, it corresponds to the optical image acquired by the camera, the image acquired by the lidar, and the image acquired by the millimeter-wave imaging device.

[0094] The process of calculating the azimuth angle of the image from a lidar and millimeter-wave imaging device is similar to the process of calculating the azimuth angle of the image from a camera, as described earlier, and will not be repeated here.

[0095] Similar to camera imaging, for lidar and millimeter-wave imaging devices, the image of a person at the current moment can be determined based on the distance difference between the center point A of a target (e.g., a person) at the current moment and the center point A at the previous moment, matching the image of the person at the previous moment. Center point A can represent the target, such as a person, or a part of the target, such as a part of a person. In the case, the target can consist of multiple parts, thus having multiple center points. Each center point is treated as a target separately, with the specific processing being the same as for a single center point; for simplicity, this setting and discussion are omitted. For the actual production environment of a mine, when calculating the probability of the existence of an unmatched target image, two factors are introduced: the change in distance and azimuth angle relative to the center point of the observed target in the previous frame (the target image at the previous moment). These factors follow a normal distribution, and the probability of the target's existence is calculated comprehensively.

[0096] In embodiments of the present invention, when a camera, lidar, and / or millimeter-wave imaging device are simultaneously installed on a vehicle, multiple imaging devices acquire images (e.g., the aforementioned first, second, and third images) at each moment. Each image includes multiple target images of multiple targets. Over time, a large number of target images are continuously generated, requiring processing of this target image information. According to the present invention, the target images are sorted. A sorting strategy for the target images to be matched can be specified, and images formed by the camera, lidar, and / or millimeter-wave imaging device are added to the sorting queue sequentially according to the type of imaging device. In one embodiment of the present invention, configurable target images can be sorted according to environmental factors, and a sorting strategy can be set, which can increase the flexibility of the target matching process. In one embodiment of the present invention, the target image set is alternately added to the sorting queue according to the type of measuring device (camera, lidar, or millimeter-wave imaging device) to which it belongs, improving the accuracy of target image matching.

[0097] For example, in situations where there is a lot of dust in the mining area, during the matching process, the target images acquired by the lidar are included in the set of unmatched target images for further target image judgment processing, while the target images acquired by the camera and millimeter-wave radar are included in the target images to be observed, thereby improving the speed and efficiency of interference elimination.

[0098] Based on the target image set tracked in the previous moment, the optimal matching is performed on the target image set to be sorted in the sorting queue order. The sum of the azimuth change value and the distance difference between the center point is selected as the parameter of the matching cost matrix, and the Hungarian algorithm is used for matching.

[0099] The cost matrix is ​​a well-known matrix used to find the minimum value of a function. In this invention, the cost matrix is ​​a matrix of cost parameters used to associate all target images acquired at the current time (i.e., one frame of data at the current time) with all target images acquired at the previous time (i.e., one frame of data at the previous time) for each pair of target selection. Specifically, each vertical row corresponds to each target at the current time, and each horizontal column corresponds to each target at the previous time. The element value in the matrix is ​​the sum of the change in azimuth angle and the distance to the center point of the target at the current time and the target at the previous time.

[0100] The Hungarian algorithm is a well-known algorithm for solving matching problems, such as a special type of bipartite graph, which is divided into two parts, and the matching relationships between points within each part. For example, the matching relationships between multiple target images at a previous time step and a subsequent time step have a maximum and a minimum number of matches, where König's theorem holds, that is, the maximum number of matches in a bipartite graph is equal to the minimum number of point covers in the graph. Specific details about the Hungarian algorithm are not described here; this invention applies the Hungarian algorithm to the cost matrix.

[0101] Before selecting a target image set to be sorted, the difference between the timestamp of the target image set and the current timestamp can be compared to ensure that the timestamp of the selected target image set is within the set threshold.

[0102] For the sorting strategy of the target images to be matched, please refer to... Figure 4 To illustrate.

[0103] First, the target image set to be sorted is obtained. Based on the sensor type (cameras are optical sensors, LiDAR is a laser sensor, and millimeter-wave imaging devices are millimeter-wave sensors), the images are added to ordered queues of each type. The ordered queues are sorted by timestamp.

[0104] Acquire target image sets (including a first target image set, a second target image set, and a third target image set) with the latest timestamps from three types of sensors: cameras, lidar, and millimeter-wave imaging devices.

[0105] Iterate through the timestamps of the target images obtained by the three imaging devices in sequence, subtract them from the current timestamp, and determine whether the result is less than the set threshold.

[0106] If the difference between the timestamp and the current time is less than the set threshold, add it to the matching queue;

[0107] After the traversal is complete, a new batch of target image sets with the latest timestamps, including the first target image set, the second target image set, and the third target image set, are added.

[0108] Subsequently, a set of target images to be sorted is obtained from the queue, and optimal matching is performed between the target image set from the previous time step and the Hungarian algorithm. The parameters of the cost matrix are the distance difference and azimuth variation difference between the center points of adjacent time steps. For example, the difference in azimuth angle between the images formed by the camera at the current time step and the image formed at the previous time step, and / or the distance difference between the center point positions of each of the multiple targets at the current time step and the image formed at the previous time step, and / or the sum of both, are used as parameters of the cost matrix. (Refer to...) Figure 5 Description matching.

[0109] First, a set of target images to be matched can be obtained from the queue of images to be matched;

[0110] Referring to the target image set of the previous moment, calculate the center point distance and azimuth change value of each target image pair (compare the image of each target with the images of all other targets in the previous moment) in the cost matrix (the calculated value is used as the element of the cost matrix). Use the sum of the two results as the parameter of the cost matrix and construct a cost matrix from the calculated values.

[0111] The Hungarian algorithm is used to perform optimal matching calculations, such as using minimum matching to select the target image corresponding to the minimum result.

[0112] Based on the optimal matching calculation results, the targets are classified and processed. These include matched targets, targets to be observed, and unmatched targets. Targets to be observed include the following categories: (1) unmatched target images in the target image set of the previous moment and whose tracking count (the number of times they return to the matching queue for rematching) does not exceed 5 times; (2) newly added target images from cameras and millimeter-wave imaging devices. Unmatched target images include the following categories: (1) unmatched target images in the target image set of the previous moment and whose tracking count exceeds 5 times; (2) newly added targets from lidar but unmatched target images.

[0113] The optimal matching calculation is now complete.

[0114] For unmatched target images, probability calculation can be used. The probability calculation for unmatched target images mainly involves calculating the probability of their existence for newly added but unmatched target images from the previous frame of the target image set to be observed.

[0115] Specific operation reference Figure 6 The following is a brief description.

[0116] Iterate through the previous frame (including, but not necessarily all, the images acquired by the camera, the LiDAR, and the millimeter-wave imaging device from the previous moment) of the target image set, and calculate and evaluate the distance (adjacent moments) and azimuth change between the target image and the center point of each target image;

[0117] Obtain the distances between all center points of the target pair in this matching process, and add the distances to each target to be observed. Based on the expected value and variance of the target distances determined by historical data records, determine the corresponding normal distribution probability density, and calculate the probability p of the distance between the center points of the target pair in this matching process.

[0118] Obtain all azimuth angle changes of the target image to be matched, and take into account the azimuth angle changes of each target image to be observed (i.e., the corresponding one). Determine the expected value and variance of the azimuth angle changes of the target image (i.e., the corresponding one) based on historical data records, and then determine the normal distribution probability density. Calculate the probability q of the azimuth angle change value corresponding to the target image to be matched.

[0119] Based on the configured calculation weight values, the probability of the center point distance and the probability of azimuth change are calculated together to obtain the final probability value of the existence of a target image;

[0120] If the final probability value is greater than the set threshold, the target image is added to the tracking list; otherwise, it is deleted.

[0121] This invention proposes a method for calculating the probability of a target under unmatched conditions, which reduces the probability of a target missing match.

[0122] Another aspect of the present invention provides a method for identifying environmental targets. The method includes the following operations.

[0123] First, a first image is acquired using a first imaging device, such as a camera. This first image includes multiple images of multiple targets. These targets can be, for example, people, obstacles, vehicles, or other objects within a mining area.

[0124] In one embodiment of the present invention, a second image can also be acquired by a lidar, wherein the second image includes multiple second target images of multiple targets;

[0125] A third image can also be acquired using a millimeter-wave imaging device, wherein the third image includes multiple third target images of multiple targets.

[0126] Similar to the above embodiments, based on the matching relationship between the azimuth angle of the first target image at the current moment of each of the plurality of first target images and the azimuth angle of the first target image at the corresponding previous moment of each of the plurality of first target images, and / or based on the distance difference between the positions of the target center points of each of the plurality of targets at the previous moment and the current moment, the first target image at the current moment that matches the first target image at the previous moment of each of the plurality of targets is determined.

[0127] Subsequently, the first target image sequence information with respect to time is obtained for each of the multiple targets in order to identify each of the multiple targets;

[0128] The azimuth angle of each of the multiple first target images in a spatial coordinate system including multiple targets is calculated based on the first imaging parameters that form the first image and the position of the first target image in the first image. The calculation of the azimuth angle can refer to the foregoing embodiment.

[0129] In one embodiment, determining a first target image at the current time that matches a first target image at a previous time for each of the plurality of targets, and determining at least another target image at the current time that matches at least another target image at a previous time for each of the plurality of targets, includes:

[0130] Using the cost matrix, the Hungarian algorithm is employed to perform optimal matching to determine the first target image at the current time and at least one other target image at the current time.

[0131] The difference between the azimuth angle of the first target image at the current time and the previous time, and / or the distance difference between the center points of each of the multiple targets at the current time and the previous time, and / or the sum of both, are used as parameters of the cost matrix.

[0132] Another aspect of the present invention provides an unmanned vehicle for mining areas, comprising:

[0133] A first imaging device, such as a camera, is configured to sense a first signal in order to acquire a first image, the first image comprising a plurality of first target images of a plurality of targets, wherein an azimuth angle of each of the plurality of first target images in a spatial coordinate system comprising the plurality of targets is calculated based on first imaging parameters forming the first image and the position of the first target image in the first image.

[0134] At least another imaging device, configured to sense at least another signal to acquire at least another image, the at least another image comprising multiple at least another target images of multiple targets, wherein an azimuth angle of each of the multiple at least another target images is calculated based on another imaging parameter forming the at least another image; and

[0135] The controller is configured as follows:

[0136] Based on the matching relationship between the azimuth angle of the first target image at the current moment of each of the multiple first target images and the azimuth angle of the first target image at the previous moment of each of the multiple first target images, and / or based on the distance difference between the positions of the target center points of each of the multiple targets at the previous moment and the current moment, determine the first target image at the current moment that matches the first target image at the previous moment of each of the multiple targets.

[0137] Based on the matching relationship between the azimuth angle of the at least other target image at the current moment and the azimuth angles of the multiple first target images and / or the azimuth angles of the multiple at least other target images at the previous moment for each of the multiple targets, and / or based on the distance difference between the positions of the center points of each of the multiple targets at the current moment and the previous moment, the at least other target image at the current moment that matches the at least other target image at the previous moment for each of the multiple targets is determined, thereby obtaining the matched first target image information and at least other target image information at the current moment for each of the multiple targets; and

[0138] The first target image sequence information with respect to time is obtained for each of the multiple targets in order to identify each of the multiple targets;

[0139] The controller is also configured to enable autonomous driving by recognizing multiple targets.

[0140] The unmanned vehicles in the mining area specifically avoid multiple identified targets, thereby achieving unmanned driving.

[0141] The various embodiments disclosed herein are not intended to limit the principles or structure of this disclosure, but rather to aid in understanding the principles and technical solutions of this disclosure. Those skilled in the art, upon reading the embodiments of this disclosure, will be able to conceive of other embodiments not described herein. These embodiments not described herein should also be considered as included in this disclosure, and the embodiments of this disclosure, as well as those not described herein, can be combined to derive other implementations of this disclosure. The scope of this invention should be defined by the claims.

Claims

1. A method for matching multiple target images of multiple sensed targets, comprising: Acquire a first image, the first image comprising multiple first target images of multiple targets; and Based on the matching relationship between the azimuth angle of the first target image at the current moment of each of the plurality of first target images and the azimuth angle of the first target image at the previous moment of each of the plurality of first target images, the first target image at the current moment that matches the first target image at the previous moment of each of the plurality of targets is determined. The azimuth angle of each of the plurality of first target images in the spatial coordinate system including the plurality of targets is calculated based on the first imaging parameters that form the first image and the position of the first target image in the first image.

2. The method according to claim 1, further comprising determining a first target image at the current moment that matches a first target image at the previous moment for each of the plurality of targets, based on the distance difference between the positions of the target center points of each of the plurality of targets at the previous moment and the current moment.

3. The method according to claim 1, wherein calculating the azimuth angle of each of the plurality of first target images based on the first imaging parameters and the position of the first target image in the first image comprises: Based on the position of each of the plurality of first target images in the first image and the imaging focal length, calculate the azimuth angle of the first target image corresponding to each of the plurality of first target images; The matching relationship between the azimuth angle of the first target image at the current moment and the azimuth angle of the first target image at the previous moment is based on the change in the difference between the azimuth angle of the first target image at the current moment and the azimuth angle at the previous moment.

4. The method according to claim 3, further comprising: Acquire at least one other image, the at least one other image comprising multiple at least one other target images of multiple targets, and calculate the azimuth angle of each of the multiple at least one other target images based on another imaging parameter forming the at least one other image and the position of the at least one other target image in the at least one other image; Wherein, at least one other target image azimuth angle is a target image azimuth angle that has been transformed into the same spatial coordinate system as the first target image azimuth angle.

5. The method according to claim 4, comprising: At least one of the images comprises at least two images obtained based on signals with at least two different physical properties.

6. The method of claim 5, wherein calculating the azimuth angle of each of the plurality of at least other target images based on another imaging parameter forming at least another image and the position of at least another target image in at least another image comprises: Based on the matching relationship between the azimuth angle of the at least other target image at the current moment of each of the plurality of at least other target images and the azimuth angles of the plurality of first target images and / or the azimuth angles of the plurality of at least other target images at the previous moment, and / or based on the distance difference between the positions of the center points of each of the plurality of targets at the current moment and the previous moment, the at least other target image at the current moment that matches the at least other target image at the previous moment of each of the plurality of targets is determined, thereby obtaining the matched first target image information and at least other target image information of each of the plurality of targets at the current moment.

7. The method of claim 6, wherein determining a first target image at a current time that matches a first target image at a previous time for each of the plurality of targets, and determining at least another target image at a current time that matches at least another target image at a previous time for each of the plurality of targets, comprises: Using the cost matrix, the Hungarian algorithm is employed to perform optimal matching to determine the first target image at the current time and at least one other target image at the current time. The difference between the azimuth angle of the first target image at the current time and the previous time, and / or the distance difference between the center point positions of each of the plurality of targets at the current time and the previous time, and / or the sum of both, are used as parameters of the cost matrix.

8. The method of claim 5, wherein at least another target image includes a second target image, obtained based on a laser signal; and / or At least one other target image, including a third target image, was obtained based on millimeter-wave signals.

9. A method for identifying environmental targets, comprising: A first signal is sensed by a first device to acquire a first image, the first image including multiple first target images of multiple targets; The matching relationship between the azimuth angle of the first target image at the current moment of each of the plurality of first target images and the azimuth angle of the first target image at the corresponding previous moment of each of the plurality of first target images; as well as First target image sequence information with respect to time is obtained for each of the plurality of targets in order to identify each of the plurality of targets; The azimuth angle of each of the plurality of first target images in the spatial coordinate system including the plurality of targets is calculated based on the first imaging parameters that form the first image and the position of the first target image in the first image.

10. The method of claim 9, further comprising determining a first target image at the current moment that matches a first target image at the previous moment for each of the plurality of targets, based on the distance difference between the positions of the target center points of each of the plurality of targets at the previous moment and the current moment.

11. The method of claim 10, wherein calculating the azimuth angle of each of the plurality of first target images based on the first imaging parameters and the position of the first target image in the first image comprises: Based on the position of each of the plurality of first target images in the first image and the imaging focal length, calculate the azimuth angle of the first target image corresponding to each of the plurality of first target images; Among them, the matching relationship between the azimuth angle of the first target image at the current moment and the azimuth angle of the first target image at the previous moment is based on the changing pattern of the difference between the azimuth angle of the first target image at the current moment and the azimuth angle at the previous moment.

12. The method of claim 11, further comprising: Acquire at least one other image, the at least one other image comprising multiple at least one other target images of multiple targets, and calculate the azimuth angle of each of the multiple at least one other target images based on another imaging parameter forming the at least one other image and the position of the at least one other target image in the at least one other image; and Based on the matching relationship between the azimuth angle of the at least other target image at the current time of each of the plurality of at least other target images and the azimuth angles of the plurality of first target images and / or the azimuth angles of the plurality of at least other target images at the previous time, and / or based on the distance difference between the positions of the center points of each of the plurality of targets at the current time and the previous time, the at least other target image at the current time that matches the at least other target image at the previous time of each of the plurality of targets is determined, thereby obtaining the matched first target image information and at least other target image information at the current time of each of the plurality of targets; Wherein, at least one other target image azimuth angle is a target image azimuth angle that has been transformed into the same spatial coordinate system as the first target image azimuth angle.

13. The method of claim 12, wherein determining a first target image at a current time that matches a first target image at a previous time for each of the plurality of targets, and determining at least another target image at a current time that matches at least another target image at a previous time for each of the plurality of targets, comprises: Using the cost matrix, the Hungarian algorithm is employed to perform optimal matching to determine the first target image at the current time and at least one other target image at the current time. The difference between the azimuth angle of the first target image at the current time and the previous time, and / or the distance difference between the center points of each of the plurality of targets at the current time and the previous time, and / or the sum of both, are used as parameters of the cost matrix.

14. The method of claim 12, wherein at least another target image includes a second target image, obtained based on a laser signal; and / or At least one other target image, including a third target image, was obtained based on millimeter-wave signals.

15. An unmanned vehicle for mining areas, comprising: A first imaging device is configured to sense a first signal in order to acquire a first image, the first image comprising a plurality of first target images of a plurality of targets, wherein a first target image azimuth angle corresponding to each of the plurality of first target images in a spatial coordinate system including the plurality of targets is calculated based on first imaging parameters forming the first image and the position of the first target image in the first image; At least another imaging device, configured to sense at least another signal in order to acquire at least another image, the at least another image comprising at least another target images of a plurality of targets, wherein an at least another target image azimuth angle corresponding to each of the plurality of at least another target images is calculated based on another imaging parameter forming the at least another image; as well as Controller, the controller is configured to: Based on the matching relationship between the azimuth angle of the first target image at the current moment of each of the plurality of first target images and the azimuth angle of the first target image at the corresponding previous moment of each of the plurality of first target images, and / or based on the distance difference between the positions of the target center points of each of the plurality of targets at the previous moment and the current moment, determine the first target image at the current moment that matches the first target image at the previous moment of each of the plurality of targets. Based on the matching relationship between the azimuth angle of the at least other target image at the current time of each of the plurality of at least other target images and the azimuth angles of the plurality of first target images and / or the azimuth angles of the plurality of at least other target images at the previous time, and / or based on the distance difference between the positions of the center points of each of the plurality of targets at the current time and the previous time, the at least other target image at the current time that matches the at least other target image at the previous time of each of the plurality of targets is determined, thereby obtaining the matched first target image information and at least other target image information at the current time of each of the plurality of targets; as well as First target image sequence information with respect to time is obtained for each of the plurality of targets in order to identify each of the plurality of targets; The controller is further configured to enable autonomous driving by recognizing the multiple targets.

Images