An elongated flexible object detection method, a terminal device and a storage medium

Abstract

The application relates to an elongated flexible object detection method, a terminal device and a storage medium, and the method comprises the following steps: collecting an elongated flexible object image and constructing a training set; constructing an elongated flexible object detection model, and training the model through the training set; after an image is input and effective feature layers are obtained through a feature extraction network, a candidate frame is generated through a candidate frame generation network and is subjected to normalization processing to obtain a normalized candidate frame; the relationship features between instances in different candidate frames are calculated through a multi-instance relationship weighting network; whether the relationship exists between the instances in different candidate frames can be judged through the relationship features; a bounding box prediction network predicts the bounding box and the category of the object in the bounding box according to the relationship features between the instances in different candidate frames; and the elongated flexible object in the image is detected through the trained model. The application can be applied to an intelligent monitoring system to effectively identify elongated flexible objects.

Description

Technical Field

[0001] This invention relates to the field of target detection, and more particularly to a method, terminal device and storage medium for detecting slender and flexible objects. Background Technology

[0002] Slender, flexible objects are very common in real-world scenarios. For example, in video surveillance of workers in industries such as power, construction, and mining, it is often necessary to detect whether workers are wearing safety belts or climbing over fences or netting. The safety belts and netting in these scenarios are slender, flexible objects. We define slender, flexible objects as objects with extreme aspect ratios and dynamically variable geometric shapes.

[0003] Instance segmentation is a technique used to predict the category and bounding box of objects in an image or video and perform pixel-level segmentation. Currently, the most advanced methods detect objects by regressing horizontal bounding boxes. For targets with regular aspect ratios (e.g., cars, airplanes, basketballs, water glasses, etc.), horizontal bounding boxes can effectively define the object's boundaries. However, slender, flexible objects are often overlooked by current detection algorithms, and most algorithms experience a significant drop in accuracy when detecting such objects.

[0004] Therefore, the detection of slender, flexible objects remains largely an unexplored field. There are currently three major challenges in detecting slender, flexible objects:

[0005] (1) Horizontal bounding boxes cannot provide accurate object appearance and boundary information. The current mainstream detection method uses horizontal bounding box annotation, which can easily and conveniently define a specified target object. However, due to the dynamic and variable geometric appearance of slender and flexible objects, such as irregular shape and easy bending or folding, the horizontal bounding box of slender objects will contain more background pixels, making the classification of slender objects more difficult than their localization.

[0006] (2) The calculation and evaluation metrics for IoU (Intersection-over-Union) are inconsistent. Object detection algorithms transform the object detection problem into a regression problem between the ground truth bounding box and the predicted bounding box. In the testing phase, IoU has been widely used as an evaluation metric to assess the detection performance of the detector. However, in the training phase, L1, L2, or smoothed L1 distance loss is commonly used as the regression loss to evaluate the distance between the ground truth bounding box and the predicted bounding box. This results in a weak correlation between the loss functions in the training and testing phases.

[0007] (3) It does not consider the correlation features between objects. Traditional implementation segmentation performs object classification and bounding box regression independently for each candidate box, without taking into account the spatial relationship between candidate boxes. However, the correlation features between objects are an important feature that helps in object recognition. Summary of the Invention

[0008] To address the aforementioned problems, this invention proposes a method for detecting slender, flexible objects, a terminal device, and a storage medium.

[0009] The specific plan is as follows:

[0010] A method for detecting slender, flexible objects includes the following steps:

[0011] S1: Collect images of slender, flexible objects and label them; construct a training set based on the labeled images.

[0012] S2: Construct a slender, flexible object detection model and train the model using a training set;

[0013] The model includes a feature extraction network, a candidate box generation network, a multi-instance relation weighting network, and a bounding box prediction network;

[0014] After the input image is processed by the feature extraction network to obtain the effective feature layer, the candidate box generation network generates candidate boxes and normalizes them to obtain normalized candidate boxes. The multi-instance relationship weighting network calculates the relationship features between instances in different candidate boxes. The relationship features can be used to determine whether there is a relationship between instances in different candidate boxes. The bounding box prediction network predicts the bounding box and the category of the object in the bounding box based on the relationship features between instances in different candidate boxes.

[0015] S3: Detect slender, flexible objects in images using the trained model.

[0016] Furthermore, the feature extraction network structure includes a ResNet101 network and a feature pyramid network. After the ResNet101 network extracts features from the input image and generates a feature map, the feature pyramid network downsamples and fuses the feature map multiple times to obtain four effective feature layers after two, three, four, and five compressions, respectively.

[0017] Furthermore, the candidate box generation network consists of a region proposal network and an ROI Align layer. The effective feature layer output by the feature extraction network is input into the region proposal network to generate candidate boxes, and then the generated candidate boxes are input into the ROI Align layer for normalization.

[0018] Furthermore, the formula for calculating the relation features of each instance output by the multi-instance relation-weighted network is as follows:

[0019]

[0020] Among them, f n Represents the relational features of the nth instance. This represents the appearance characteristics of the nth instance; the Concat function represents concatenation. Indicates the Nth r The weighted property of the nth instance in a multi-instance relation weighted module, N r This is a hyperparameter representing the number of multi-instance relationship weighting modules accessed in the multi-instance relationship weighted network;

[0021] The weighted property f of the nth instance R The formula for calculating (n) is:

[0022]

[0023] Where m and n represent the m-th and n-th instances in the instance set input to the multi-instance relation weighted network, respectively, ω mn This represents the relationship weight between the m-th instance and the n-th instance. W represents the appearance features of the m-th instance. V Represents a linear transformation matrix;

[0024] The relationship weight ω between the m-th instance and the n-th instance mn The calculation formula is:

[0025]

[0026] in, This represents the geometric weights between the m-th instance and the n-th instance. Let represent the appearance weights of the m-th instance and the n-th instance, exp represent the exponential function with base e, and k represent the instance index. The geometric weights of the k-th instance and the n-th instance, This represents the appearance weight between the k-th instance and the n-th instance;

[0027] The appearance weights of the m-th instance and the n-th instance The calculation formula is:

[0028]

[0029] Where dot represents the dot product operation. W represents the appearance features of the nth instance. K W QThey respectively represent the following: and The matrix projected onto the lower-dimensional space, d k Indicates the feature dimension after projection;

[0030] Geometric weights of the m-th instance and the n-th instance The calculation formula is:

[0031]

[0032] Where max represents finding the maximum value. Let ε represent the geometric features of the m-th instance and the n-th instance, respectively. G Functional representation embeds geometric features into a high-dimensional space using cosine and sine functions, W G This represents the transformation of geometric features in a high-dimensional space into a scalar matrix.

[0033] Furthermore, the bounding box prediction network predicts the loss function L for the bounding box. reg for:

[0034]

[0035] Where A represents the ground truth bounding box, B represents the predicted bounding box, C represents the minimum containment box of the ground truth bounding box and the predicted bounding box, and IoU represents the intersection-union ratio between bounding boxes A and B.

[0036] A terminal device for detecting slender, flexible objects includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described above in the embodiments of the present invention.

[0037] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above in the embodiments of the present invention.

[0038] The present invention adopts the above technical solution, which can be applied to intelligent monitoring systems to effectively identify slender and flexible objects, achieve high detection accuracy, and effectively improve the efficiency of safety monitoring of workers. Attached Figure Description

[0039] Figure 1 The diagram shown is a flowchart of Embodiment 1 of the present invention.

[0040] Figure 2 The image shown is a schematic diagram of the image annotation in this embodiment.

[0041] Figure 3The diagram shown is a schematic of the network structure of the slender flexible object detection model in this embodiment.

[0042] Figure 4 The diagram shown is a schematic of the network structure of the region proposal network in this embodiment.

[0043] Figure 5 The diagram shown is a schematic diagram of the network structure of the multi-instance relationship weighted network in this embodiment.

[0044] Figure 6 The diagram shown is a schematic representation of the network structure of the multi-instance relationship weighting module in this embodiment.

[0045] Figure 7 The diagram shown illustrates the mask generation process in this embodiment.

[0046] Figure 8 The diagram shown illustrates the calculation process of the generalized overlap ratio in this embodiment.

[0047] Figure 9 The diagram shown illustrates the calculation process of Mask IoU in this embodiment. Detailed Implementation

[0048] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention.

[0049] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0050] Example 1:

[0051] This invention provides a method for detecting slender, flexible objects, such as... Figure 1 As shown, the method includes the following steps:

[0052] S1: Collect images of slender, flexible objects and label them. Construct a training set based on the labeled images.

[0053] In this embodiment, images of slender, flexible objects were collected from a company, including images of safety belts, fences, and workers. The images can be divided into two categories. The first category consists of 5250 images taken by on-site inspectors using cameras. These images have a higher resolution, measuring 5184×3888 pixels, and are typically taken from a horizontal or upward angle relative to the target object. The second category consists of 1300 images captured from power grid monitoring cameras. These images have relatively smaller pixels (1280×720 or 1920×1080 pixels), and are taken from a downward angle relative to the target object.

[0054] To improve the accuracy of subsequent model training, this embodiment also includes data cleaning of the acquired images. Data cleaning mainly consists of two parts: first, removing images irrelevant to the slender, flexible object; second, classifying the remaining images and further removing those with severe target occlusion or excessively small target sizes that affect recognition. After data cleaning, 1499 images were obtained that met the requirements.

[0055] Furthermore, since the pixel sizes of the acquired images vary, the input images need to be normalized to a uniform size during model training to facilitate model computation. If the image sizes are the same, multiple images can be combined into a single batch during input, improving GPU utilization. Therefore, this embodiment also performs image cropping and scaling to unify the image size (specifically, normalizing it to 1280×800 pixels), reducing the time spent on image preprocessing during training.

[0056] Image annotation is a process of drawing labels on an image. A polygonal bounding box is drawn around a detected object and labeled with the object's category. The polygonal bounding box defines the object's location in the image. This embodiment uses the latest interactive segmentation and automatic annotation tool, EISeg, to annotate the image. EISeg achieves semi-automatic annotation through interactive segmentation technology based on region seeds. It first pre-annotates the image using a pre-trained model. For areas with inaccurate or erroneous annotations, it then uses a series of green dots (foreground) and red dots (background) to precisely adjust the edges of the target object, thereby achieving refined annotation. Figure 2 As shown.

[0057] In this embodiment, after image cleaning and annotation, a total of 2368 images of slender, flexible objects were obtained, categorized into three types: human, seine, and safety belt. The dataset was divided into training, validation, and test sets, with proportions of 80%, 19%, and 1%, respectively.

[0058] S2: Construct a slender, flexible object detection model and train the model using a training set.

[0059] The network structure of the slender flexible object detection model is as follows: Figure 3 As shown, it includes, in sequence, a feature extraction network, a candidate box generation network, a multi-instance relationship weighting network, and a bounding box prediction network.

[0060] (1) The input image is processed by a feature extraction network to extract effective feature layers. Since the ResNet neural network has achieved excellent classification results on the ImageNet dataset, ResNet101 is used as the backbone network of the feature extraction network in this embodiment to extract features of the input image and obtain feature maps. After ResNet101, the feature maps are downsampled and fused multiple times by a Feature Pyramid Network (FPN) to obtain four effective feature layers after two, three, four and five compressions respectively.

[0061] (2) The structure of the candidate box generation network includes a region proposal network (RPN) and an ROI Align layer. After the effective feature layer output by the feature extraction network is fed into the region proposal network, the candidate box generation network generates candidate boxes and performs normalization processing to obtain normalized candidate boxes.

[0062] like Figure 4 As shown, the Region Proposal Network (RPN) is a lightweight network that uses a sliding window to traverse all candidate boxes on the feature map and search for potential regions where objects may exist, thus generating candidate boxes. Each candidate box has three sizes, and each size has three different aspect ratios (e.g., 5×7 and 7×5), resulting in a total of nine different candidate boxes. Each feature map covers approximately 200,000 candidate boxes. After the effective feature layer output from the feature extraction network is fed into the RPN, the RPN scans all candidate boxes to predict k candidate boxes. Each candidate box includes four coordinate information (x, y, w, h) and two classification categories (foreground or background).

[0063] The obtained candidate boxes are fed into the ROI Align layer for normalization. That is, RoI Align maps candidate boxes of different sizes generated by RPN onto a feature map of uniform size. RoI Align uses bilinear interpolation to obtain the floating-point coordinates of pixels, which can effectively preserve the pixel features of the target edges, thereby obtaining more accurate edge segmentation results.

[0064] (3) The multi-instance relationship weighted network is used to calculate the relationship features between instances in different candidate boxes. The normalized candidate boxes output by the multi-candidate box generation network are input into the multi-instance relationship weighted network, which outputs the relationship features between instances in each candidate box.

[0065] like Figure 5 As shown, the relation features of each instance output by the multi-instance relation weighted network are N. r After concatenating multiple instance relationship weighted modules together, and... The cumulative result is calculated using the following formula:

[0066]

[0067] Among them, f n Represents the relational features of the nth instance. This represents the appearance characteristics of the nth instance; the Concat function represents concatenation. Indicates the Nth r The weighted property of the nth instance in a multi-instance relation weighted module, N r This is a hyperparameter representing the number of multi-instance relationship weighting modules accessed in the multi-instance relationship weighting network. In this embodiment, N is set to... r =16.

[0068] The multi-instance relationship weighting module is used to extract the appearance features f of each instance in the candidate box. A and geometric features f G And proposes an instance relation weighting function f R (n) Models the spatial relationships between instances in the candidate box, such as Figure 6 As shown.

[0069] Most instance detection methods that rely on object relationships treat the determination of these relationships as a post-processing step. After detecting the coordinates of the objects, they calculate the geometric relationships between them (such as the distance between their centroids) to determine their relevance. However, this method incurs additional computational overhead and cannot be trained end-to-end on deep neural networks. The inter-instance relationship features are crucial; for example, when detecting workers in a construction scene, objects like safety belts and safety nets are more likely to be detected in the image. To enable the neural network to learn the inter-instance relationship features, this embodiment introduces a multi-instance relationship weighting module, which considers not only the geometric features f of each instance... G It also takes into account the appearance characteristics of each instance. A Furthermore, the weighted module treats all instances in the image as a group of mutually influential objects as input for recognition, with the relationship weight between instances being ω. mn Instead of treating instances in an image as individual entities.

[0070] In this embodiment, the multi-instance relationship weighting module is designed based on a self-attention mechanism. The attention mechanism can build a dependency model for each element in the sequence without considering the distance between elements. The input to the attention module includes elements of dimension d. k queries and keys, and dimension d v The values ​​are calculated by performing a dot product operation between a query and all keys to obtain the relevance coefficient α between the query and the keys. Then, the softmax function is applied to obtain the weights of the values. Given a query q, all keys are packed into matrix K, and the values ​​are packed into matrix V. The output value is the weighted average of the input values, that is, the output v contains the relationship information between q and K.

[0071]

[0072] Suppose an instance includes geometric features f G and appearance features f A f G It is a four-dimensional object bounding box containing the object's bounding box coordinates (Xmin, Ymin, Xmax, Ymax), where (Xmin, Ymin) represent the coordinates of the top-left corner of the bounding box, and (Xmax, Ymax) represent the coordinates of the bottom-right corner of the bounding box; f A These are the features of an instance, which can be obtained through a convolutional neural network.

[0073] Given an input set containing N instances Regarding the nth instance, the weighted property f of the entire instance set R The formula for calculating (n) is as follows:

[0074]

[0075] Among them, W V It is a linear transformation matrix used for... Performing a linear transformation is equivalent to V in formula (1). The relationship weight ω between the m-th instance and the n-th instance... mn The formula for calculating the influence from other objects is as follows:

[0076]

[0077] Relationship weights consist of appearance weights and geometry weights. The formulas for calculating the appearance weights of the m-th instance and the n-th instance are as follows:

[0078]

[0079] Among them, W K and W Q Equivalent to K and Q in formula (1), it will change the original features and They are projected into a low-dimensional space to calculate their correlation; the projected feature dimension is d. k dimension.

[0080] The formula for calculating the geometric weights of the m-th instance and the n-th instance is as follows:

[0081]

[0082] The calculation of geometric weights involves two steps. First, the geometric features of the instance are embedded into a high-dimensional space using cosine and sine functions, denoted as ε. G The second step is to use matrix W to represent the geometric features in the high-dimensional space. G Convert to a scalar.

[0083] A multi-instance relation weighted network is added after the candidate box generation network. The candidate box generation network predicts candidate boxes for potential regions in the image by scanning the feature map of the input image. The multi-instance relation weighted network improves the performance of object detection by calculating the contextual relationships between candidate boxes.

[0084] (4) The bounding box prediction network predicts the bounding box and the category of the object in the bounding box based on the relationship features between instances in different candidate boxes (i.e., the output of the multi-instance relationship weighted network).

[0085] Furthermore, this embodiment also includes segmenting the predicted bounding box through a mask branch to obtain a pixel-level mask for the object, thereby achieving more refined detection of slender and flexible objects.

[0086] The bounding box prediction network includes: feeding the adjusted local feature layer into the classification and regression model, which mainly has two branches: (1) the classification branch determines whether the candidate box actually contains the object; (2) the bounding box regression branch is used to regress the predicted bounding box, and the candidate box is adjusted to become the predicted box, thus completing the object detection. Then it enters the mask semantic segmentation branch, which uses the previously obtained local feature layer containing the object to perform semantic segmentation, thereby improving the prediction speed of the overall model. The mask generation process is as follows: Figure 7 As shown.

[0087] Object detection and instance segmentation algorithms transform the object detection problem into a regression problem between the ground truth bounding boxes and the predicted bounding boxes. To address the inconsistency between the objective function and the evaluation metric, the objective function can be changed to a generalized overlap ratio loss function, thus unifying the two. The proposed formula for calculating the generalized overlap ratio is as follows:

[0088]

[0089] Where R(A,B) represents the overlap ratio between bounding boxes A and B, A represents the ground truth bounding box, B represents the predicted bounding box, C represents the minimum containment box of the ground truth bounding box and the predicted bounding box, and IoU represents the intersection-union ratio between bounding boxes A and B.

[0090] The generalized overlap ratio (GUPR) exhibits scale invariance. When rectangles A and B intersect, the GUPR shows a strong correlation with the objective function; even when rectangles A and B do not intersect, the distance between A and B can still be calculated. Furthermore, unlike IoU, which only focuses on the overlapping region, the GUPR considers not only the overlapping region but also other non-overlapping regions (d1 and d2), such as... Figure 8 As shown, it can better reflect the overlap between bounding boxes and is more sensitive to the bounding box offset of slender objects.

[0091] The improved generalized overlap ratio bounding box regression loss function is defined as:

[0092] L reg =L R =1-R(A,B)

[0093] Instance segmentation algorithms use the Mask IoU metric to measure the quality of the masks generated by instance segmentation. For example... Figure 9 As shown, the dark polygonal boxes are the manually labeled ground truth bounding boxes, and the light-colored polygonal boxes are the predicted bounding boxes generated by the model. The Mask IoU value is the ratio of the intersection to the union of the ground truth polygonal box region G and the predicted polygonal box region P, as shown in the following formula:

[0094]

[0095] Setting a Mask IoU threshold can determine the accuracy of the model's predictions. When the calculated Mask IoU value exceeds the threshold, it means that the predicted box has hit the ground truth box. When the Mask IoU value is less than or equal to the threshold, it means that the target was missed or the ground truth box may not have been hit.

[0096] Instance segmentation uses mean average precision (mAP) as the evaluation metric. AP represents the average precision of a single class, while mAP represents the mean average precision across all classes. AP50 and AP75 represent the mAP measurements when the IoU thresholds are 0.5 and 0.75, respectively. S AP M and AP L These represent the mAP measurements of bounding boxes with pixel areas less than 322, between 322 and 962, and greater than 962 pixels, respectively.

[0097] The loss function in model training is used to evaluate the difference between the model's predicted output and the ground truth. A smaller loss function indicates that the predicted output is closer to reality. The model's loss function can be divided into two parts: the first part is the loss of the RPN network, which generates candidate boxes and adjusts their bounding boxes, thus including the RPN foreground / background classification loss L. RPN_cls and RPN target bounding box regression loss L RPN_reg The second part is network loss, including classification loss L. cls Bounding box regression loss L reg and pixel segmentation loss L mask And the Mask IoU network loss, i.e., the prediction mask crossover ratio loss L maskiou The model's total loss function is the sum of all individual losses:

[0098] L = L RPN +L cls +L reg +L mask +L maskiou

[0099] The loss function of the RPN network is as follows:

[0100]

[0101] The RPN network uses a binary cross-entropy loss function for classification. i p represents the predicted probability that the object in the i-th anchor is the target object. i *This indicates that the object in the i-th anchor is the true label of the target object. When the object to be detected is in the i-th anchor, p i * If p is 1, then p is 1. i * The value is 0. Since the RPN is only responsible for detecting the presence of targets rather than classifying them, each anchor's class is either 0 or 1. i t represents the predicted bounding box. i * Indicates the bounding box of the annotation. N obj The number of minibatches is set to 256. N reg The number of anchors is set to 2400. The classification loss and bounding box regression loss of the RPN network are determined by N. obj and N reg Normalization. Due to N obj and N reg Since the difference in quantity between the two loss functions is too large, a hyperparameter λ is introduced to balance the influence between the two loss functions. Let the value of λ be 10.

[0102] Classification loss function L cls The formula is as follows:

[0103]

[0104] Improved generalized overlap ratio bounding box regression loss function L reg The formula is:

[0105] L reg =L R =1-R(A,B)

[0106] L mask It predicts the loss of the mask:

[0107]

[0108] L maskiou It is the loss for predicting the cross-union ratio of the mask, using mean squared error to calculate the regression loss between the predicted mask and its matched ground truth.

[0109] S3: Detect slender, flexible objects in images using the trained model.

[0110] The embodiments of the present invention have the following beneficial effects.

[0111] (1) For slender and flexible objects, the instance segmentation method can not only locate and classify each target in the image, but also perform pixel-level segmentation of the target to achieve more refined detection.

[0112] (2) The bounding box regression loss is improved from the smooth L1 loss to the generalized overlap ratio loss function proposed in this invention, so that the training objective function and the evaluation function are unified.

[0113] (3) Design a multi-instance relationship weighting module in the instance segmentation network. First, extract the appearance and geometric features of each instance through a deep convolutional neural network. Then, learn the features of the mutual relationships between all targets in the image end-to-end through the multi-instance relationship weighting module, which improves the accuracy of instance segmentation, especially the accuracy of small targets.

[0114] Example 2:

[0115] The present invention also provides a terminal device for detecting slender flexible objects, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the method embodiment described above in Embodiment 1 of the present invention.

[0116] Furthermore, as an executable solution, the elongated flexible object detection terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The elongated flexible object detection terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described structure of the elongated flexible object detection terminal device is merely an example and does not constitute a limitation on the device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the elongated flexible object detection terminal device may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0117] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the slender flexible object detection terminal device, connecting all parts of the device via various interfaces and lines.

[0118] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the slender flexible object detection terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0119] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0120] If the modules / units integrated in the slender flexible object detection terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0121] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for detecting slender, flexible objects, characterized in that, Includes the following steps: S1: Collect images of slender, flexible objects and label them; construct a training set based on the labeled images. S2: Construct a slender, flexible object detection model and train the model using a training set; The model includes a feature extraction network, a candidate box generation network, a multi-instance relation weighting network, and a bounding box prediction network; After the input image is processed by the feature extraction network to obtain the effective feature layer, the candidate box generation network generates candidate boxes and normalizes them to obtain normalized candidate boxes. The multi-instance relationship weighting network calculates the relationship features between instances in different candidate boxes. The relationship features can be used to determine whether there is a relationship between instances in different candidate boxes. The bounding box prediction network predicts the bounding box and the category of the object in the bounding box based on the relationship features between instances in different candidate boxes. S3: Detect slender, flexible objects in images using the trained model; The formula for calculating the relation features of each instance output by the multi-instance relation weighted network is as follows: Among them, f n Represents the relational features of the nth instance. This represents the appearance characteristics of the nth instance; the Concat function represents concatenation. Indicates the Nth r The weighted property of the nth instance in a multi-instance relation weighted module, N r This is a hyperparameter representing the number of multi-instance relationship weighting modules accessed in the multi-instance relationship weighted network; The weighted property f of the nth instance R The formula for calculating (n) is: Where m and n represent the m-th and n-th instances in the instance set input to the multi-instance relation weighted network, respectively, ω mn This represents the relationship weight between the m-th instance and the n-th instance. W represents the appearance features of the m-th instance. V Represents a linear transformation matrix; The relationship weight ω between the m-th instance and the n-th instance mn The calculation formula is: in, This represents the geometric weights between the m-th instance and the n-th instance. Let represent the appearance weights of the m-th instance and the n-th instance, exp represent the exponential function with base e, and k represent the instance index. The geometric weights of the k-th instance and the n-th instance, This represents the appearance weight between the k-th instance and the n-th instance; The appearance weights of the m-th instance and the n-th instance The calculation formula is: Where dot represents the dot product operation. W represents the appearance features of the nth instance. K W Q They respectively represent the following: and The matrix projected onto the lower-dimensional space, d k Indicates the feature dimension after projection; Geometric weights of the m-th instance and the n-th instance The calculation formula is: Where max represents finding the maximum value. Let ε represent the geometric features of the m-th instance and the n-th instance, respectively. G Functional representation embeds geometric features into a high-dimensional space using cosine and sine functions, W G This represents the transformation of geometric features in a high-dimensional space into a scalar matrix.

2. The method for detecting slender, flexible objects according to claim 1, characterized in that: The structure of the feature extraction network includes a ResNet101 network and a feature pyramid network. After the ResNet101 network extracts the features of the input image and generates a feature map, the feature pyramid network downsamples and fuses the feature map multiple times to obtain four effective feature layers after two, three, four, and five compressions, respectively.

3. The method for detecting slender, flexible objects according to claim 1, characterized in that: The candidate box generation network consists of a region proposal network and an ROI Align layer. The effective feature layer output by the feature extraction network is input into the region proposal network to generate candidate boxes, and then the generated candidate boxes are input into the ROI Align layer for normalization.

4. The method for detecting slender, flexible objects according to claim 1, characterized in that: The loss function L of the bounding box prediction network predicts the bounding boxes. reg for: Where A represents the ground truth bounding box, B represents the predicted bounding box, C represents the minimum containment box of the ground truth bounding box and the predicted bounding box, and IoU represents the intersection-union ratio between bounding boxes A and B.

5. A slender, flexible object detection terminal device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images