Image target detection method and ore detection positioning tracking method using the same

Abstract

The application relates to an image target detection method and an ore detection positioning tracking method applying the same. In view of the problem that the target size gap is large and it is difficult to detect all, the method is based on an improved YOLOv5-lite image target detection method, different features are bidirectionally fused through the introduction of a weight mechanism in the middle layer network of YOLOv5-lite, single-class target detection tasks can be better performed, the ability of a model to extract multi-scale target features is enhanced, and detection efficiency and detection precision are improved.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to an image target detection method and its application in ore detection, positioning and tracking methods. Background Technology

[0002] Thanks to the improved feature extraction capabilities of convolutional neural networks and the widespread use of mobile devices, many lightweight convolutional neural networks, such as YOLOv5-lite, have been developed for object detection, suitable for embedded devices, mobile devices, and edge computing. However, when there is a large size difference between targets, these lightweight networks often struggle to detect them completely, thus affecting the detection results. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] The main objective of this invention is to propose an image target detection method and its application in ore detection, positioning, and tracking, which can improve detection efficiency and accuracy.

[0005] To achieve the above objectives, a first aspect of the present invention provides an image target detection method, which includes the following steps:

[0006] Multiple feature maps of different resolutions are extracted from the image to be detected based on the backbone network. These feature maps are then input into an improved intermediate layer network of YOLOv5-Lite for feature enhancement, resulting in multiple enhanced feature maps output by the intermediate layer network. The improved intermediate layer network comprises multiple sub-networks with the same number of layers as the backbone network outputs at different resolutions. The top and bottom sub-networks each include input and output nodes, while each of the remaining sub-networks includes input, intermediate, and output nodes. The multi-layer sub-network includes a first path, a second path, and a third path. The first path is the path from the input node to the corresponding output node in each sub-network. The second path is a top-down path from the input node of the top sub-network to the intermediate node of the next sub-network, and so on, until the intermediate node of the sub-network above the bottom sub-network is reached to the output node of the bottom sub-network. The third path is the path from the bottom up from each output node to the top output node. Furthermore, the improved intermediate layer network adds a corresponding learnable weight to each input feature map of different resolutions.

[0007] The detector determines several target detection boxes corresponding to each of the enhanced feature maps, and selects the target detection box with the highest confidence from all target detection boxes corresponding to all the enhanced feature maps according to the non-maximum suppression method, and identifies the target according to the target detection box with the highest confidence; wherein, the target is a pre-specified object to be detected in the image to be detected.

[0008] This application provides an image target detection method that addresses the problem of difficulty in detecting all targets with large size differences. This method is based on the improved YOLOv5-lite image target detection method. It introduces a weight mechanism to bidirectionally fuse different features across scales on the basis of the intermediate layer network of YOLOv5-lite, which can better perform single-class target detection tasks, enhance the model's ability to extract multi-scale target features, and improve detection efficiency and accuracy.

[0009] In some embodiments, the step of extracting multiple feature maps of different resolutions of the image to be detected based on the backbone network includes:

[0010] The image to be detected is subjected to a convolution and max pooling layer with a stride of 2, respectively, and downsampling is performed once through each convolution and max pooling layer. Then, it is downsampled again through a separable convolution with a stride of 2, and after downsampling, feature extraction is performed through three separable convolution layers with a stride of 1 to obtain a third resolution feature map.

[0011] The third resolution feature map is processed by a separable convolution with a stride of 2. Downsampling is performed, and then features are extracted using seven separable convolutional layers with a stride of 1 to obtain a second-resolution feature map.

[0012] The second resolution feature map is processed by a separable convolution with a stride of 2. Downsampling is performed, and then features are extracted using three layers of separable convolutions with a stride of 1 to obtain the first-resolution feature map.

[0013] In some embodiments, the improved intermediate layer network comprises three sub-networks, and the improved intermediate layer network outputs enhanced feature maps in the following manner:

[0014] The first resolution feature map After convolution and upsampling, it is compared with the second resolution feature map. The features are concatenated, and then the concatenated feature map is input into the first C3 module to obtain the intermediate feature map.

[0015] The intermediate feature map After convolution and upsampling, it is compared with the third resolution feature map. The concatenated feature map is then input into the second C3 module to obtain the third resolution feature map. Corresponding enhanced feature map

[0016] The third resolution feature map corresponds to the enhanced feature map. After convolution, it is combined with the intermediate feature map. Second resolution feature map The concatenated feature map is then input into the third C3 module to obtain the second resolution feature map. Corresponding enhanced feature map

[0017] The second resolution feature map Corresponding enhanced feature map After convolution, it is compared with the feature map at the first resolution. The features are then concatenated, and the resulting feature map is input into the fourth C3 module to obtain the first resolution feature map. Corresponding enhanced feature map

[0018] In some embodiments, the intermediate feature map is generated using the following formula:

[0019]

[0020] in, The C3(·) function performs a convolution operation using the C3 module, the ConV(·) function performs the convolution operation, and ω1 and ω2 are learnable weights. This is the second resolution feature map. Let ξ1 be the first resolution feature map, and let Resize(·) be a linear interpolation upsampling operation. ξ1 is a constant.

[0021] In some embodiments, the second resolution feature map Corresponding enhanced feature map Generate using the following formula:

[0022]

[0023] Where ω3, ω4, and ω5 are learnable weights, the C3(·) function performs convolution using the C3 module, and the ConV(·) function performs the convolution operation. For the third resolution feature map For the enhanced feature map, ξ2 is a constant.

[0024] To achieve the above objectives, a second aspect of the present invention provides a method for detecting, locating, and tracking ore, the method comprising the following steps:

[0025] Acquire a series of multiple frames of images to be detected captured by a monocular camera;

[0026] Based on the image target detection method described in the first aspect above, the minerals in each of the images to be detected are detected;

[0027] Track the ore to determine the two-dimensional coordinates of the ore's center point in the pixel coordinate system;

[0028] The monocular camera is calibrated to obtain its intrinsic and extrinsic parameters. Based on these parameters, the transformation relationship between the pixel coordinate system and the world coordinate system is determined. According to the transformation relationship, the two-dimensional coordinates of the pixel coordinate system are transformed into two-dimensional coordinates in the world coordinate system.

[0029] Extract the length and width of the circumscribed rectangle of the ore, and take the shorter side of the length and width as the height of the ore. Based on the length and width of the circumscribed rectangle, the two-dimensional coordinates of the world coordinate system, and the height, construct the ore into a cube in the world coordinate system to control the gripping device to grip the cube; wherein the base of the gripping device is set at the origin of the world coordinate system.

[0030] This application provides a method for ore detection, localization, and tracking, based on an improved YOLOv5-lite image target detection method, for identifying and locating the position of ore in images. Addressing the issue of large size variations in ore, the intermediate layer network of YOLOv5-lite is improved by introducing a weight mechanism to bidirectionally fuse different features across scales. This enables better single-class ore detection tasks, enhances the model's ability to extract multi-scale ore features, and improves detection efficiency and accuracy. Furthermore, this method utilizes a camera calibration scheme and investigates the structural characteristics of the ore itself to transform the 2D ore detection results into the 3D real world, reducing the cost of detection equipment while ensuring high detection accuracy.

[0031] In some embodiments, the intrinsic and extrinsic parameters of the monocular camera are obtained in the following manner:

[0032] A checkerboard calibration plate is arranged; wherein the checkerboard calibration plate can cover the field of view of the monocular camera;

[0033] The calibration images of the checkerboard calibration board at multiple different positions are captured by the monocular camera.

[0034] The intrinsic and extrinsic parameters of the monocular camera are calculated using the Zhang Zhengyou calibration method based on the checkerboard corner points in the calibration image.

[0035] In some embodiments, the ore is tracked according to the DeepSORT target tracking algorithm.

[0036] To achieve the above objectives, a third aspect of the present invention provides an electronic device, comprising: at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the above-described image target detection method and / or the above-described ore detection, positioning, and tracking method.

[0037] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described image target detection method and / or the above-described ore detection, positioning and tracking method. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic flowchart of an image target detection method provided in one embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the structure of PANet provided in one embodiment of this application;

[0041] Figure 3 This is a schematic diagram of a multi-layer sub-network structure of an improved intermediate layer network provided in one embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the network structure used in an image target detection method provided in one embodiment of this application;

[0043] Figure 5 yes Figure 1 A detailed flowchart of step S101 is shown below;

[0044] Figure 6 yes Figure 1 A detailed flowchart of step S102 is shown below;

[0045] Figure 7 This is a schematic diagram of the three-layer sub-network structure of the improved intermediate layer network provided in one embodiment of this application;

[0046] Figure 8 This is a flowchart illustrating an embodiment of the ore detection, positioning, and tracking method provided in this application.

[0047] Figure 9 This is a flowchart illustrating DeepSORT provided in one embodiment of this application;

[0048] Figure 10 This is a target tracking result diagram provided in one embodiment of this application;

[0049] Figure 11 This is a flowchart illustrating the process of obtaining the intrinsic and extrinsic parameters of a monocular camera according to one embodiment of this application;

[0050] Figure 12 This application provides a chessboard design drawing and a physical image of one embodiment of the present application;

[0051] Figure 13 This application provides a chessboard calibration diagram and a chessboard corner point identification diagram in one embodiment.

[0052] Figure 14 This is a schematic diagram illustrating the transformation relationship between various coordinates provided in one embodiment of this application;

[0053] Figure 15 This is a schematic diagram of the height expansion of the two-dimensional coordinates of an ore according to an embodiment of this application;

[0054] Figure 16 This is a diagram showing the corner positions of a chessboard grid according to one embodiment of this application;

[0055] Figure 17 This is a schematic diagram of the training process loss provided in one embodiment of this application;

[0056] Figure 18 This is a schematic diagram of ore testing results provided in one embodiment of this application;

[0057] Figure 19 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] First Embodiment

[0062] Reference Figure 1 One embodiment of this application provides an image target detection method, which includes the following steps S101 to S103:

[0063] Step S101: Extract multiple feature maps of different resolutions from the image to be detected based on the backbone network.

[0064] Step S102: Input multiple feature maps of different resolutions into the improved intermediate layer network of YOLOv5-Lite for feature enhancement, resulting in multiple enhanced feature maps output by the intermediate layer network. The improved intermediate layer network includes multiple sub-networks with the same number of layers as the backbone network output resolution. The top and bottom sub-networks in the multi-layer sub-networks include input nodes and output nodes, while each of the remaining sub-networks includes input nodes, intermediate nodes, and output nodes. The multi-layer sub-networks include a first path, a second path, and a third path. The first path is the path from the input node to the corresponding output node of each sub-network. The second path is the path from top to bottom from the input node of the top sub-network to the intermediate node of the next sub-network, and then to the intermediate node of the next sub-network, until the intermediate node of the sub-network above the bottom sub-network is to the output node of the bottom sub-network. The third path is the path from bottom to top of each output node. Furthermore, the improved intermediate layer network adds a corresponding learnable weight to the input feature maps of different resolutions.

[0065] Step S103: Determine several target detection boxes corresponding to each enhanced feature map based on the detector, and select the target detection box with the highest confidence from all target detection boxes in all enhanced feature maps according to the non-maximum suppression method, and identify the target based on the target detection box with the highest confidence; wherein, the target is the object to be detected in the image to be detected in advance.

[0066] In this embodiment, the image to be detected is the image in which the target needs to be detected. YOLOv5-lite is a network for target detection, which includes a backbone network, intermediate layers, detectors, and a post-processing module. The backbone network can extract feature maps of multiple different resolutions from the image. Therefore, in this embodiment, feature maps at different resolutions (i.e., multiple scales) can be extracted from the image to be detected using the YOLOv5-lite backbone network.

[0067] In the intermediate layers of YOLOv5-Lite, YOLOv5-Lite uses an FPN+PAN structure. YOLOv5Lite avoids the repeated use of the C3 module and high-channel C3 modules. The C3 module is an improved version of the CSPBottleneck proposed by the YOLOv5 authors; it is simpler, faster, and lighter, achieving better results with nearly similar loss. However, the C3 module uses multi-path split convolution. Testing has shown that frequent use of the C3 module and high-channel C3 modules consumes more cache space and reduces running speed. Therefore, the repeated use of the C3 module and high-channel C3 modules should be avoided in the design process. YOLOv5-Lite performs channel pruning on the YOLOv5 head, referencing the design principles of ShuffleNet v2, and also improves the FPN+PAN (i.e., PANet) structure in YOLOv5. This structure is as follows... Figure 2 As shown, however, this structure does not perform well in detecting difficult targets such as edges, blur, and occlusions. In step S102, this method improves the intermediate layer network of YOLOv5-Lite, replacing the original PANet structure, and introduces a weighting mechanism to fuse different features bidirectionally across scales, thereby enhancing the ability to extract features of multi-scale image targets.

[0068] This application improves upon the YOLOv5-Lite intermediate layer network by adding a bidirectional cross-scale weighting mechanism and employing a weighted bidirectional fusion approach. This enhances the feature information of feature maps at different scales, fusing as many features as possible without significantly increasing computational cost, thus improving the detection performance of difficult targets such as edges, blurriness, and occlusion. (See reference...) Figure 3There are three main paths in this network: one is a top-down path, such as from the top-level input node to the intermediate node of the next layer, then to the intermediate node of the next layer, and so on, until the penultimate intermediate node reaches the bottom-level output node. This path transmits high-level semantic information (beneficial for object classification); another is a bottom-up path, transmitting low-level positional information (beneficial for object localization); and a third is an additional path between input and output nodes within the same layer, such as from the top-level input node to the output node, and from the input node and intermediate node of the next layer to the output node. Furthermore, because different input features have different resolutions, their contributions to the output features are often uneven. To address this issue, this embodiment adds an extra weight to each input feature in the intermediate layer network, allowing the network to learn the importance of each input feature. The weighting formula is as follows:

[0069]

[0070] Where w i ≥0 is achieved by each w i ReLU is then applied to ensure this, and ∈ = 0.0001 is a very small value to avoid numerical instability. The value of each normalized weight falls between 0 and 1.

[0071] like Figure 2 and Figure 3 Compared to the PANet structure, this example removes nodes with only one input or output edge from the intermediate layer network structure of the conventional YOLOv5-Lite. This is because these nodes do not significantly improve the overall feature fusion performance; removing them reduces computation and simplifies the bidirectional network. Secondly, the intermediate layer network structure in this embodiment adds an additional path between the input and output nodes, enabling the fusion of more features across layers and improving detection accuracy.

[0072] Reference Figure 4 , Figure 5 In a preferred embodiment, step S101 includes the following steps S1011 to S1013:

[0073] Step S1011: Perform a convolution and max pooling layer with a stride of 2 on the image to be detected, and perform downsampling once through each of the convolution and max pooling layers; then perform downsampling again through a separable convolution layer with a stride of 2, and finally perform feature extraction through three separable convolution layers with a stride of 1 after downsampling to obtain the third resolution feature map.

[0074] Step S1012: Apply a separable convolution with a stride of 2 to the third resolution feature map. Downsampling is performed, and then features are extracted using seven separable convolutional layers with a stride of 1 to obtain a second-resolution feature map.

[0075] Step S1013: Apply a separable convolution with a stride of 2 to the second resolution feature map. Downsampling is performed, and then features are extracted using three layers of separable convolutions with a stride of 1 to obtain the first-resolution feature map.

[0076] Compared to the backbone network of YOLOv5-lite, the YOLOv5-lite backbone network includes Shuffle Net V2, a lightweight convolutional neural network that offers better performance and a smaller model size. The main idea of ​​Shuffle Net V2 is to group the channels of the feature maps according to a certain pattern, shuffle the channels of different groups, and then perform convolution operations. This approach is often limited by the computing power of the device. Therefore, to reduce computational overhead, the method improves the backbone network in step S101 by redesigning the number of channels in each layer of the feature maps in Shuffle Net V2. Specifically, the number of channels in the third-resolution feature map, the second-resolution feature map, and the first-resolution feature map are reduced to 32, 64, and 128, respectively, further improving computational speed. Figure 3 In the diagram, the third resolution feature map, the second resolution feature map, and the first resolution feature map correspond to the following in the figure: and

[0077] Reference Figure 4 , Figure 6 and Figure 7 In some embodiments of this application, the intermediate layer network in step S102 includes three sub-networks, and the intermediate layer network outputs multiple enhanced feature maps in the following manner:

[0078] Step S1021: Transfer the first resolution feature map After convolution and upsampling, it is compared with the second resolution feature map. The features are concatenated, and then the concatenated feature map is input into the first C3 module to obtain the intermediate feature map.

[0079] Step S1022: Transfer the intermediate feature map After convolution and upsampling, it is compared with the third resolution feature map. The concatenated feature map is then input into the second C3 module to obtain the third-resolution feature map. Corresponding enhanced feature map

[0080] Step S1023: Match the third resolution feature map with the enhanced feature map. After convolution, it is combined with the intermediate feature map Second resolution feature map The concatenated feature map is then input into the third C3 module to obtain the second-resolution feature map. Corresponding enhanced feature map

[0081] Step S1024: Transfer the second resolution feature map Corresponding enhanced feature map After convolution, it is compared with the first resolution feature map. The feature maps are then concatenated, and the concatenated feature maps are input into the fourth C3 module to obtain the first resolution feature map. Corresponding enhanced feature map

[0082] Specifically, the three feature inputs with different resolutions obtained through different stages of the backbone network (such as...) Figure 4 In and Then, the features with different resolutions obtained from different stages of the backbone network were combined. and As input, it is fed into the intermediate layer network, as referenced. Figure 4 and Figure 7 The second column of the intermediate layer, formed by the convolutional layer of the second layer, creates a pathway connecting the upper and lower layers. This pathway will... The features input to this layer are fused and output to the next convolutional layer and the path of the third column. The first and third layers do not have convolutional layers because there can only be one output or one input at this position. This does not significantly improve the overall fusion effect but will increase the amount of computation.

[0083] The intermediate feature map is generated using the following formula:

[0084]

[0085] in, The C3(·) function performs a convolution operation using the C3 module, the ConV(·) function performs the convolution operation, and ω1 and ω2 are learnable weights. This is the second resolution feature map. The first resolution feature map is generated using the Resize(·) function, which performs linear interpolation upsampling with ξ1 as a constant of 0.0001. The second resolution feature map, corresponding to the enhanced feature map, is generated using the following formula:

[0086]

[0087] Where ω3, ω4, and ω5 are learnable weights. For the third resolution feature map For the enhanced feature map, ξ2 is a constant with a value of 0.0001.

[0088] Reference Figure 4 In step S103, the detector is responsible for generating detection results using the feature maps output from the intermediate layer. The detector includes a classification head, a bounding box head, and a target confidence head. The classification head predicts the target category, the bounding box head predicts the target's location and size, and the target confidence head predicts the target's confidence score. Finally, the non-maximum suppression (NMS) algorithm is used to remove duplicate target detection boxes and retain the target detection box with the highest confidence score.

[0089] The specific process of post-processing based on nonmaximum suppression is as follows:

[0090] Step 1: Sort all bounding boxes according to their confidence level;

[0091] The second step is to extract and save the detection box with the highest confidence level, calculate the C-IOU between this box and other boxes, and delete the detection boxes with C-IOU greater than the set threshold.

[0092] Third, repeat steps one and two until the confidence level is less than the set threshold or there are no detection boxes.

[0093] This embodiment addresses the problem of difficulty in detecting targets with large size differences. Based on the improved YOLOv5-lite image target detection method, this embodiment introduces a weight mechanism to bidirectionally fuse different features across scales on the basis of the intermediate layer network of YOLOv5-lite. This enables better single-class target detection tasks, enhances the model's ability to extract multi-scale target features, and improves detection efficiency and accuracy.

[0094] This embodiment also addresses the issue of target detection being often limited by the computing power of mobile devices by further lightweighting the ShuffleNetv2 backbone, reducing network computational consumption, and making the network suitable for scenarios with limited computing power, such as embedded devices, mobile devices, and edge devices. Furthermore, this embodiment also sets up a corresponding three-layer sub-network for the improved backbone network, further reducing consumption.

[0095] Second Embodiment

[0096] As a vital pillar of the national economy, the mining industry urgently needs to improve its level of intelligent manufacturing to enhance production efficiency and reduce costs. Generally, the ore production process includes key stages such as exploration, mining, crushing, and screening. Among these, ore screening, a crucial step, aims to classify raw ore, separating ores of different types, sizes, and grades to improve processing efficiency and beneficiation effectiveness. Currently, tungsten ore screening primarily employs two methods: manual screening and ore separators. Manual screening requires workers to select quartzite containing wolframite and scheelite from large quantities of waste rock to extract tungsten ore. However, this method requires a significant workforce, often requiring eight workers on a single ore screening line. Ore separators, on the other hand, separate ore based on physical properties such as density and electromagnetic properties, but they are primarily suitable for processing small particles and have limited effectiveness for large ore pieces. Therefore, they cannot be applied to the separation of large-particle ores.

[0097] Currently, the use of convolutional neural networks for automatic ore sorting is mainly for ores with small particle sizes. However, the size difference between different ores is large. Using conventional convolutional neural networks will not only affect the ore detection effect, but will also be limited by the computing power of the detection equipment.

[0098] Reference Figure 8 One embodiment of this application provides a method for detecting, locating, and tracking ore. This method includes the following steps S201 to S205, as detailed below:

[0099] Step S201: Acquire a series of multiple frames of images to be detected captured by a monocular camera.

[0100] In step S201, a monocular camera is first deployed, then a video stream of the ore is captured using the monocular camera, and multiple frames of images to be detected are extracted from it. Simultaneously, the pixel coordinate system corresponding to the monocular camera is determined.

[0101] Step S202: Detect the minerals in each image to be detected based on the image target detection method.

[0102] Here, the method of the first embodiment described above is used to detect the location of the ore. Specifically, it is used... Figure 3 The network shown is used for testing.

[0103] Step S203: Track the ore to determine the two-dimensional coordinates of the ore's center point in the pixel coordinate system.

[0104] After the target detection of the ore is achieved through step S202, the real-time position of the ore in multiple frames of the image to be detected can be obtained. However, at this time, only the position information of the ore can be obtained, and the identity information and future position prediction information of the ore cannot be obtained. Therefore, if only simple target detection is performed and only the position information in each frame is relied upon, the precise grasping of the robotic arm cannot be achieved.

[0105] Therefore, to obtain the identity information and predicted future location of each detected ore in the images captured by the monocular camera, after object detection identifies all targets in multiple frames of images to be detected, the DeepSORT object tracking algorithm is used to predict the identity and motion information of all ore. DeepSORT extracts the features of all targets and matches them with previously tracked targets to determine if it belongs to one of the previously tracked targets. If a match is successful, the state of the tracked target is updated; otherwise, it is treated as a new target and a new tracker is created for it. The entire process is as follows: Figure 9 As shown. During the tracking process, DeepSORT uses a Kalman filter to predict the target's position and velocity, and adjusts the predictions based on the actual observed position. Since the DeepSORT target tracking algorithm is common knowledge, it will not be described in detail here. The target tracking results are as follows. Figure 10 As shown, Figure 10 Two ores were detected on the left. The right side of the attached image shows the prediction of the ores' motion information. Compared to the simple position information in object detection, the right side of the attached image shows that object detection can assign a unique ID to all ores in multiple frames of images to be detected and predict the motion information of the ores.

[0106] Step S204: Calibrate the monocular camera and obtain its intrinsic and extrinsic parameters. Determine the transformation relationship between the pixel coordinate system and the world coordinate system based on the intrinsic and extrinsic parameters. Based on the transformation relationship, convert the two-dimensional coordinates of the pixel coordinates into two-dimensional coordinates in the world coordinate system.

[0107] Reference Figure 11 In this embodiment, the intrinsic and extrinsic parameters of the monocular camera are obtained in the following manner:

[0108] Step S2041: Obtain a chessboard calibration board; wherein the chessboard calibration board can cover the field of view of the monocular camera.

[0109] Step S2042: Take calibration images of the checkerboard calibration board at multiple different positions using a monocular camera.

[0110] Step S2043: Based on the checkerboard corner points in the calibration image, calculate the intrinsic and extrinsic parameters of the monocular camera using the Zhang Zhengyou calibration method.

[0111] Step S205: Extract the length and width of the circumscribed rectangle of the ore, and take the shorter side of the length and width as the height of the ore. Based on the length and width of the circumscribed rectangle, the two-dimensional coordinates and height in the world coordinate system, construct the ore into a cube in the world coordinate system to control the gripping device to grip the cube; wherein, the base of the gripping device is set at the origin of the world coordinate system.

[0112] The following is a detailed introduction:

[0113] Based on the height and field of view of the monocular camera, design a camera calibration board that can cover its field of view, such as... Figure 12 As shown ( Figure 12 The right side of the attached diagram shows the actual calibration plate, and the left side is a schematic diagram. In the diagram, each small square in the checkerboard is 20 mm long and wide, and the entire checkerboard is 440 mm long and 300 mm wide. Due to the ease of fabrication and high precision of the checkerboard, a rectangular checkerboard calibration plate was chosen. Each checkerboard square is 20 mm long, with 15 rows and 22 columns, and is attached to a 1 cm thick rectangular wooden board for camera calibration.

[0114] By capturing images of the calibration board at multiple locations using a monocular camera, detecting the checkerboard corner points in the images, and employing Zhang Zhengyou's calibration method to obtain the camera's intrinsic and extrinsic parameters, such as... Figure 13 As shown.

[0115] Once the camera's intrinsic and extrinsic parameters are obtained, the camera coordinate system o-uv and the world coordinate system O can be determined. W -X W Y W Z W The conversion relationship between them is as follows: Figure 14 As shown in Figure (a) is the pixel coordinate system, Figure (b) is the world coordinate system, and Figure (c) is the robot arm coordinate system), the robot arm base is placed at the world coordinate origin O. W At this point, assuming the height of the robotic arm's base coordinate relative to the base is h, then the robotic arm's base coordinate (X) is... r ,Y r Z r ) can be described as (X W ,Y W In summary, the ore coordinates in the pixel coordinates can be converted into the grasping coordinates in the robotic arm coordinate system (there is a height difference h between the robotic arm coordinate system and the world coordinate system).

[0116] When the position of the ore in the image is obtained through target detection using a monocular camera, and converted from the pixel coordinate system to the robotic arm coordinate system, the resulting coordinates of the ore's center position are still two-dimensional coordinates (x, y, y). m ,ym The robotic arm lacks the height position information (x) required for grasping. m ,y m ,z m Generally, the height of an object is measured using LiDAR or a depth camera. However, using LiDAR is expensive and requires fusion measurement of LiDAR and a monocular camera. Furthermore, it may cause misidentification due to the close proximity of the objects.

[0117] Reference Figure 15 Based on the data collected from the ore production site, it was found that the height and width of the ore are approximately the same. Therefore, to reduce equipment costs and computational complexity, the ore is considered as a regular cube. When the bounding rectangle (x, y) of the ore is detected in a two-dimensional image using the aforementioned image target detection method, as shown... Figure 15 As shown, the shorter side will be used as the height information of the ore, that is, the ore will be regarded as a three-dimensional cube with (x, y, x), as... Figure 15 As shown on the right, the three-dimensional coordinate information of the ore is now obtained, which can then be used in conjunction with the robotic arm for grasping.

[0118] This embodiment has the following advantages:

[0119] To address the challenges of detecting large variations in ore size and the limitations of computing power in such equipment, this embodiment utilizes an improved YOLOv5-lite image target detection method to identify and locate ore positions within images. To further address the issue of large ore size variations, this embodiment improves the intermediate layer network of YOLOv5-lite by introducing a weight mechanism for bidirectional cross-scale feature fusion. This enhances the model's ability to extract multi-scale ore features, improving detection efficiency and accuracy. To address the stringent computational requirements, this embodiment further lightweights the shufflenetv2 backbone, reducing computational costs and making the network suitable for scenarios with limited computing power, such as embedded devices, mobile devices, and edge computing. Furthermore, by employing a camera calibration scheme and investigating the structural characteristics of the ore itself, this embodiment transforms the 2D ore detection results into the 3D real world, reducing equipment costs while maintaining high detection accuracy.

[0120] Third Embodiment

[0121] This section provides a set of coordinate transformation examples.

[0122] The above embodiments mentioned converting coordinates in the pixel coordinate system to coordinates in the world coordinate system. The accuracy of the coordinate transformation determines the accuracy of the robotic arm's grasping position; therefore, high-precision coordinate transformation is very important. Here, we will verify the transformation accuracy by performing coordinate transformation on the corner coordinates of the chessboard grid.

[0123] Assume the origin of the world coordinate system is at point O. W If four points A, B, C, and D are selected on the chessboard, their positions in world coordinates are (6,6), (6,16), (16,6), and (16,16), in mm.

[0124] Reference Figure 16 In the pixel coordinate system, the coordinates of these four points in the image are also known, so their positions in the world coordinate system can be obtained using a calculation formula. The transformation results are shown in the table below:

[0125]

[0126]

[0127] As can be seen from the table, the results obtained by converting pixel coordinates to world coordinates using a monocular camera have an error of less than 0.5mm compared with the actual results in world coordinates, whether in terms of horizontal or vertical distance calculation, which meets the requirements of high precision.

[0128] A set of object detection examples are also provided here;

[0129] A large amount of video footage of ore was captured from an actual industrial site using a monocular camera, directly beneath factory lighting. After acquisition, the video was segmented, frames extracted, and the data cleaned manually to remove invalid data. Then, the labelme tool was used to manually label all the data, selecting the associated tungsten ore quartzite to be identified. This quartzite was then divided into training, validation, and test sets to complete the dataset construction. Detailed information about the dataset is shown in the table below.

[0130] Dataset Total frames training set test set Validation set resolution Tungsten Ore Dataset 1700 1360 170 170 2448×2048

[0131] Data augmentation is a common method to increase the size and quality of training datasets. Therefore, after the dataset collection is completed, data augmentation methods such as flipping, stretching and compressing, and rotating can be used to increase the amount of data and improve the robustness of the model.

[0132] Target detection part (e.g.) Figure 4The network was trained using the SGD optimizer with an initial learning rate of 1e-2. A cosine periodic adjustment was used to adjust the learning rate, with a minimum learning rate of 1e-4. The optimizer's internal momentum parameter was set to 0.937, and the weight decay coefficient was set to 5e-4. The number of training iterations was set to 90, and data augmentation was applied to the dataset to improve the model's recognition accuracy.

[0133] Training process loss as Figure 17 As shown.

[0134] Training results show that the object detection network achieved an AP (Achievement of Progress) of 91.23% for ore detection, with a Precision of 88.74% and a Recall of 88.74%, and an F1 score of 0.89. Some ore detection results are shown below. Figure 18 As shown.

[0135] Fourth embodiment

[0136] Reference Figure 19 This application also provides an electronic device, which includes:

[0137] At least one memory;

[0138] At least one processor;

[0139] At least one program;

[0140] The program is stored in memory, and the processor executes at least one program to implement the image target detection method or ore detection, positioning and tracking method described above in this disclosure.

[0141] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0142] The electronic devices according to embodiments of this application will now be described in detail.

[0143] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0144] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the image target detection method or ore detection, positioning, and tracking method of the embodiments of this invention.

[0145] The input / output interface 1800 is used to implement information input and output.

[0146] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0147] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0148] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0149] Fifth embodiment

[0150] This invention also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described image target detection method or ore detection, positioning and tracking method.

[0151] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] The embodiments described in this invention are intended to more clearly illustrate the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0153] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0154] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0156] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0157] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0161] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. An image target detection method, characterized by, The image target detection method comprises the following steps: According to the backbone network, a plurality of feature maps of different resolutions of the to-be-detected image are extracted; the extraction of three feature maps of different resolutions of the to-be-detected image according to the backbone network comprises: The image to be detected is subjected to a convolution and max pooling layer with a stride of 2, respectively, and downsampling is performed once through each convolution and max pooling layer. Then, it is downsampled again through a separable convolution with a stride of 2, and after downsampling, feature extraction is performed through three separable convolution layers with a stride of 1 to obtain a third resolution feature map. The third resolution feature map is processed by a separable convolution with a stride of 2. Downsampling is performed, and then features are extracted using seven separable convolutional layers with a stride of 1 to obtain a second-resolution feature map. The second resolution feature map is processed by a separable convolution with a stride of 2. Downsampling is performed, and then features are extracted using three layers of separable convolutions with a stride of 1 to obtain the first-resolution feature map. ; The plurality of feature maps of different resolutions are input into an improved intermediate layer network of YOLOv5-Lite for feature enhancement, to obtain a plurality of enhanced feature maps output by the intermediate layer network, wherein the improved intermediate layer network comprises a plurality of layers of sub-networks with the same number of layers as the number of resolutions output by the backbone network, the top layer of sub-networks and the bottom layer of sub-networks in the plurality of layers of sub-networks comprise input nodes and output nodes, and each remaining layer of sub-networks comprises input nodes, intermediate nodes and output nodes, the plurality of layers of sub-networks comprise a first path, a second path and a third path, the first path is a path from the input node to the corresponding output node of each layer of sub-networks; the second path is a path from the input node of the top layer of sub-networks to the intermediate node of the next layer of sub-networks of the top layer of sub-networks, and then to the intermediate node of the next layer, and finally from the intermediate node of the upper layer of sub-networks of the bottom layer of sub-networks to the output node of the bottom layer of sub-networks from top to bottom; the third path is a path from each output node to the top layer output node from bottom to top; and the improved intermediate layer network adds a corresponding learnable weight for the input feature maps of different resolutions; the improved intermediate layer network comprises three layers of sub-networks, and outputs the enhanced feature maps in the following manner: The first resolution feature map is input into a first C3 module to obtain a first intermediate feature map After convolution and up-sampling, the second resolution feature map is input into a second C3 module to obtain a second intermediate feature map The first intermediate feature map and the second intermediate feature map are spliced, and the spliced feature map is input into a first C3 module to obtain an intermediate feature map ; The intermediate feature map is input into a second C3 module to obtain a third resolution feature map After convolution and up-sampling, the third resolution feature map is input into a third C3 module to obtain a fourth resolution feature map Splicing, and inputting the spliced feature map into a second C3 module to obtain the third resolution feature map The corresponding enhanced feature map ; The third resolution feature map corresponds to the enhanced feature map After convolution, the intermediate feature map is concatenated with the second resolution feature map The second resolution feature map The second resolution feature map is obtained by concatenating the feature maps obtained by the first C3 module and the second C3 module The third resolution feature map corresponds to the enhanced feature map ; the second resolution feature map the corresponding enhanced feature map after convolution, the first resolution feature map splicing, and then inputting the spliced feature map to a fourth C3 module to obtain the first resolution feature map the corresponding enhanced feature map ; According to the detector, a plurality of target detection boxes corresponding to each of the enhanced feature maps are determined, a target detection box with the highest confidence is selected from all target detection boxes corresponding to all enhanced feature maps according to a non-maximum suppression method, and a target is identified according to the target detection box with the highest confidence; wherein the target is a to-be-detected object in the to-be-detected image.

2. The image target detection method of claim 1, wherein, The intermediate feature map is generated by the following formula: ， wherein, is an intermediate feature map, is a convolution operation using a C3 module, is a convolution operation, is a learnable weight, is a second resolution feature map, is a first resolution feature map, is a linear interpolation up-sampling operation, is a constant.

3. The image target detection method of claim 2, wherein, the second resolution feature map corresponding enhanced feature map generated by the following equation: ， wherein, is a learnable weight, is a convolution operation using the C3 module, is a convolution operation, is a third resolution feature map corresponds to the enhanced feature map, is a constant.

4. A method of ore detection, localization and tracking, characterized by, The ore detection positioning and tracking method comprises the following steps: Obtaining a plurality of continuous frames of to-be-detected images captured by a monocular camera; Detecting the ore in each of the to-be-detected images based on the image target detection method according to any one of claims 1 to 3; Tracking the ore to determine the two-dimensional coordinates of the center point of the ore in the pixel coordinate system; Calibrating the monocular camera to obtain the internal and external parameters of the monocular camera, and determining the conversion relationship between the pixel coordinate system and the world coordinate system according to the internal and external parameters; Converting the two-dimensional coordinates of the pixel coordinate system into two-dimensional coordinates in the world coordinate system according to the conversion relationship; Extracting the length and width of the circumscribed rectangle of the ore, taking the shorter side of the length and width as the height of the ore, and constructing the ore into a cube in the world coordinate system according to the length and width of the circumscribed rectangle, the two-dimensional coordinates of the world coordinate system and the height, so as to control a grabbing device to grab the cube; wherein the base of the grabbing device is arranged at the origin of the world coordinate system.

5. The ore detection, localization, and tracking method of claim 4, wherein, The internal and external parameters of the monocular camera are obtained in the following manner: A chessboard calibration board is arranged; wherein the chessboard calibration board can cover the field of view of the monocular camera; A plurality of calibration images of the chessboard calibration board in different positions are captured according to the monocular camera; According to the corner points of the chessboard in the calibration images, the internal and external parameters of the monocular camera are calculated by using Zhang Zhengyou calibration method.

6. The ore detection, localization, and tracking method of claim 4, wherein, The ore is tracked according to the DeepSORT target tracking algorithm.

7. An electronic device, comprising: Comprise: At least one control processor and a memory connected in communication with the at least one control processor; The memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the image target detection method of any one of claims 1 to 3 and / or the ore detection positioning tracking method of any one of claims 4 to 6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to execute the image target detection method of any one of claims 1 to 3 and / or the ore detection positioning tracking method of any one of claims 4 to 6.

Images