Conical barrel detection method, device, medium and equipment

Abstract

The application discloses a detection method, device, medium and equipment for a conical barrel, comprising the following steps: based on color images and depth information of the conical barrel photographed from multiple perspectives, extracting three-dimensional features of the conical barrel corresponding to the multiple perspectives, fusing the three-dimensional features of the conical barrel to obtain fused features of the conical barrel; identifying an image to be completed containing an occluded conical barrel and a target image of the conical barrel which is not occluded in the color images; extracting a first complete conical barrel model corresponding to the target image, and establishing an initial conical barrel model according to the occluded conical barrel in the image to be completed, and completing the initial conical barrel model by using the fused features of the conical barrel to obtain a second complete conical barrel model; based on the first complete conical barrel model and the second complete conical barrel model, establishing a conical barrel recognition model; and identifying whether the conical barrel is contained in an image to be recognized and positioning the position of the conical barrel by using the conical barrel recognition model, so that the conical barrel images under multiple perspectives can be recognized, and the accuracy of the detection of the conical barrel is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium, and equipment for detecting conical barrels. Background Technology

[0002] Traffic cones, also known as traffic cones, cone-shaped road signs, or red hats, are a type of road traffic isolation and warning facility. They must be placed during road construction and maintenance or in case of unexpected malfunctions to ensure safety. Accurate detection of traffic cones is crucial for autonomous driving systems and intelligent traffic monitoring systems.

[0003] Currently, bucket detection primarily employs traditional image processing techniques or deep learning methods. Traditional image processing techniques mainly rely on features such as color, texture, and shape for identification. However, since the shape and color of buckets can vary depending on the manufacturer and usage environment, the effectiveness of traditional color- and shape-based image processing techniques is limited. For example, occluded buckets suffer from inaccurate detection results due to shape constraints. Deep learning methods, on the other hand, train models using a large number of samples to achieve automatic bucket recognition. While deep learning methods have shown good performance in handling complex scenes and backgrounds, they still struggle with occluded bucket images. Furthermore, the training samples obtained by deep learning methods under occlusion conditions are limited due to variations in the occluding object or the occlusion angle, thus leading to inaccurate detection results.

[0004] In summary, existing cone detection methods have relatively low accuracy in detecting obstructed cones. Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus, medium and equipment for detecting conical barrels, the main purpose of which is to solve the problem of low accuracy of current conical barrel detection results.

[0006] According to one aspect of this application, a method for detecting a conical barrel is provided, the method comprising:

[0007] Based on the color images and depth information of the conical barrel taken from multiple perspectives, the three-dimensional features of the conical barrel corresponding to multiple perspectives are extracted, and the three-dimensional features of the conical barrel are fused to obtain the fused features of the conical barrel.

[0008] Based on the depth information, the color image is identified to include a complete image of an occluded cone and a target image of an unoccluded cone.

[0009] Extract the first complete cone-shaped barrel model from the target image viewpoint corresponding to the target image, and establish an initial cone-shaped barrel model based on the occluded cone-shaped barrels in the image to be completed, and use the cone-shaped barrel fusion feature to complete the initial cone-shaped barrel model to obtain the second complete cone-shaped barrel model from the viewpoint of the image to be completed.

[0010] A cone-shaped barrel recognition model is established based on the first and second complete cone-shaped barrel models from multiple perspectives.

[0011] When an image to be identified is received, the cone-shaped barrel recognition model is used to identify whether the image contains a cone-shaped barrel and to locate the position of the cone-shaped barrel.

[0012] This application provides a method for detecting cone-shaped barrels. Based on color images and depth information of cone-shaped barrels taken from multiple perspectives, a cone-shaped barrel fusion feature is obtained. The method identifies the image to be completed containing occluded cone-shaped barrels and the target image of unoccluded cone-shaped barrels within the color images. Based on the image to be completed, the cone-shaped barrel fusion feature, and the target image, a cone-shaped barrel recognition model is obtained. The cone-shaped barrel recognition model accurately identifies cone-shaped barrels and determines their positions. Occluded cone-shaped barrels can also be accurately identified, greatly improving the accuracy of cone-shaped barrel detection.

[0013] Optionally, the step of extracting three-dimensional features of the cone-shaped barrel corresponding to multiple viewpoints based on color images and depth information captured from multiple perspectives, and fusing the three-dimensional features of the cone-shaped barrel to obtain fused features of the cone-shaped barrel includes:

[0014] The color image and depth information corresponding to each viewpoint are merged to obtain the comprehensive image corresponding to each viewpoint. Based on the pre-trained 3D feature extraction model, 3D features are extracted from each comprehensive image to obtain the 3D features of the cone barrel corresponding to each viewpoint.

[0015] Based on the shooting position information corresponding to each viewpoint and the position information corresponding to the reference viewpoint, calculate the viewpoint transformation matrix corresponding to each viewpoint when it is transformed into the reference viewpoint;

[0016] Based on the viewpoint transformation matrix and the three-dimensional features of the cone barrel corresponding to each viewpoint, the three-dimensional transformation features of the cone barrel corresponding to each viewpoint are obtained when each viewpoint is transformed into a reference viewpoint.

[0017] The 3D transformation features of the cone-shaped barrel corresponding to each viewpoint are fused to obtain the cone-shaped barrel fused features.

[0018] This application provides a method for detecting cone-shaped barrels. After performing viewpoint transformation on the three-dimensional features of the cone-shaped barrel corresponding to each viewpoint, the three-dimensional transformed features of the cone-shaped barrel corresponding to each viewpoint are fused to obtain the fused features of the cone-shaped barrel. This method can accurately characterize the various features of the cone-shaped barrel under multiple viewpoints, providing a strong basis for the identification of occluded cone-shaped barrels and improving the accuracy of cone-shaped barrel identification.

[0019] Optionally, the step of identifying, based on the depth information, the color image containing the image to be completed with the occluded cone and the target image with the unoccluded cone includes:

[0020] Perform object recognition on the color image to obtain an image of each object contained in the color image;

[0021] Based on the depth information corresponding to the color image, the front-to-back relationship between objects in the color image is determined;

[0022] Based on the front-to-back relationship between objects in the color image, the images of the objects, and the preset cone image, the image to be completed for the occluded cone and the target image for the unoccluded cone are determined.

[0023] This application provides a method for detecting cone-shaped objects, which accurately identifies the image to be completed containing occluded cone-shaped objects and the target image where the cone-shaped objects are not occluded in each color image based on the front-back relationship between objects in a color image, thereby improving the accuracy of cone-shaped object image recognition.

[0024] Optionally, the step of using the cone-shaped barrel fusion features to complete the initial cone-shaped barrel model to obtain a second complete cone-shaped barrel model from the perspective of the image to be completed includes:

[0025] Based on the initial cone-shaped barrel model, a predicted cone-shaped barrel image is generated;

[0026] The cone-shaped barrel fusion feature is used to adjust the predicted cone-shaped barrel image to obtain the completed cone-shaped barrel image;

[0027] Based on the completed cone-shaped barrel image, a second complete cone-shaped barrel model is extracted from the perspective of the image to be completed.

[0028] This application provides a method for detecting cone-shaped barrels. The method utilizes cone-shaped barrel fusion features to complete the image of the occluded cone-shaped barrel. Based on the completed image, a second complete cone-shaped barrel model is obtained from the perspective of the image to be completed, which improves the accuracy of the cone-shaped barrel model. Based on the second complete cone-shaped barrel model, a cone-shaped barrel recognition model is obtained, which further improves the accuracy of the cone-shaped barrel recognition model.

[0029] Optionally, extracting the first complete cone-shaped barrel model from the target image viewpoint corresponding to the target image includes:

[0030] Based on a pre-trained bucket extraction model, the target image is extracted to obtain the first complete bucket model from the target image perspective.

[0031] This application provides a method for detecting cone-shaped barrels. By extracting a cone-shaped barrel model through a pre-trained cone-shaped barrel extraction model, a first complete cone-shaped barrel model from the perspective of the target image is obtained, which improves the accuracy of the cone-shaped barrel model establishment and lays the foundation for the accurate establishment of the cone-shaped barrel recognition model. Based on the first complete cone-shaped barrel model, a cone-shaped barrel recognition model is obtained, which further improves the accuracy of the cone-shaped barrel recognition model establishment.

[0032] Optionally, the cone-shaped barrel recognition model includes a segmenter and a discriminator.

[0033] The segmenter is used to extract multiple candidate images of cones from the image to be identified, and the discriminator is used to identify the real image containing the cone from the multiple candidate images of cones and the fake images of cones.

[0034] This application provides a method for detecting cone-shaped barrels. The cone-shaped barrel recognition model integrates complete cone-shaped barrel models from multiple perspectives. Through classifiers and discriminators, it accurately identifies whether the image to be identified contains cone-shaped barrels. Even occluded cone-shaped barrels can be accurately identified, thus improving the accuracy of cone-shaped barrel detection.

[0035] Alternatively, the position of the conical barrel can be located using the following method:

[0036] After determining that the image to be identified contains a cone-shaped barrel, the depth information and shooting position information corresponding to the image to be identified are obtained;

[0037] Obtain the depth information corresponding to the cone-shaped barrel from the depth information corresponding to the image to be identified;

[0038] Based on the depth information corresponding to the cone-shaped barrel and the shooting position information, the position information of the cone-shaped barrel is determined.

[0039] This application provides a method for detecting conical barrels, which, based on the depth information and shooting position information of the conical barrel, can quickly and accurately obtain the specific position information of the conical barrel, thereby improving the accuracy and speed of conical barrel detection.

[0040] According to another aspect of this application, a detection device for a conical barrel is provided, comprising:

[0041] The feature fusion module is used to extract the three-dimensional features of the cone-shaped barrel corresponding to multiple viewpoints based on the color images and depth information of the cone-shaped barrel taken from multiple viewpoints, and to fuse the three-dimensional features of the cone-shaped barrel to obtain the fused features of the cone-shaped barrel.

[0042] The first recognition module is used to identify, based on the depth information, the color image containing the image to be completed with the occluded cone and the target image with the unoccluded cone;

[0043] The first acquisition module is used to extract the first complete cone-shaped barrel model from the target image viewpoint corresponding to the target image, and to establish an initial cone-shaped barrel model based on the occluded cone-shaped barrels in the image to be completed, and to complete the initial cone-shaped barrel model using the cone-shaped barrel fusion feature to obtain the second complete cone-shaped barrel model from the viewpoint of the image to be completed.

[0044] The second acquisition module is used to establish a cone-shaped barrel recognition model based on the first complete cone-shaped barrel model and the second complete cone-shaped barrel model from multiple perspectives.

[0045] The second recognition module is used to identify whether the image to be recognized contains a cone and to locate the position of the cone when the image to be recognized is received, by means of the cone recognition model.

[0046] This application provides a cone-shaped barrel detection device. Based on color images and depth information of the cone-shaped barrel captured from multiple perspectives, it obtains cone-shaped barrel fusion features. It identifies the to-be-completed image containing occluded cone-shaped barrels and the target image of the unoccluded cone-shaped barrels within the color images. Based on the to-be-completed image, the cone-shaped barrel fusion features, and the target image, it obtains a cone-shaped barrel recognition model. This model accurately identifies cone-shaped barrels and determines their positions, even accurately identifying occluded cone-shaped barrels, significantly improving the accuracy of cone-shaped barrel detection.

[0047] Optionally, the feature fusion module is further configured to:

[0048] The color image and depth information corresponding to each viewpoint are merged to obtain the comprehensive image corresponding to each viewpoint. Based on the pre-trained 3D feature extraction model, 3D features are extracted from each comprehensive image to obtain the 3D features of the cone barrel corresponding to each viewpoint.

[0049] Based on the shooting position information corresponding to each viewpoint and the position information corresponding to the reference viewpoint, calculate the viewpoint transformation matrix corresponding to each viewpoint when it is transformed into the reference viewpoint;

[0050] Based on the viewpoint transformation matrix and the three-dimensional features of the cone barrel corresponding to each viewpoint, the three-dimensional transformation features of the cone barrel corresponding to each viewpoint are obtained when each viewpoint is transformed into a reference viewpoint.

[0051] The 3D transformation features of the cone-shaped barrel corresponding to each viewpoint are fused to obtain the cone-shaped barrel fused features.

[0052] This application provides a cone-shaped barrel detection device that transforms the three-dimensional features of the cone-shaped barrel corresponding to each viewpoint to obtain a fused cone-shaped barrel feature after fusing the three-dimensional transformed features of the cone-shaped barrel corresponding to each viewpoint. This device can accurately characterize the various features of the cone-shaped barrel under multiple viewpoints, providing a strong basis for the identification of occluded cone-shaped barrels and improving the accuracy of cone-shaped barrel identification.

[0053] Optionally, the first identification module is used to:

[0054] Perform object recognition on the color image to obtain an image of each object contained in the color image;

[0055] Based on the depth information corresponding to the color image, the front-to-back relationship between objects in the color image is determined;

[0056] Based on the front-to-back relationship between objects in the color image, the images of the objects, and the preset cone image, the image to be completed for the occluded cone and the target image for the unoccluded cone are determined.

[0057] This application provides a cone-shaped barrel detection device that, based on the front-to-back relationship between objects in a color image, accurately identifies the image to be completed containing an occluded cone-shaped barrel and the target image of the unoccluded cone-shaped barrel in each color image, thereby improving the accuracy of cone-shaped barrel image recognition.

[0058] Optionally, the first acquisition module is further configured to:

[0059] Based on the initial cone-shaped barrel model, a predicted cone-shaped barrel image is generated;

[0060] The cone-shaped barrel fusion feature is used to adjust the predicted cone-shaped barrel image to obtain the completed cone-shaped barrel image;

[0061] Based on the completed cone-shaped barrel image, a second complete cone-shaped barrel model is extracted from the perspective of the image to be completed.

[0062] This application provides a cone-shaped barrel detection device that uses cone-shaped barrel fusion features to complete the image of the occluded cone-shaped barrel. Based on the completed image, a second complete cone-shaped barrel model is obtained from the perspective of the image to be completed, which improves the accuracy of the cone-shaped barrel model establishment. Based on the second complete cone-shaped barrel model, a cone-shaped barrel recognition model is obtained, which further improves the accuracy of the cone-shaped barrel recognition model establishment.

[0063] Optionally, the first acquisition module is further configured to:

[0064] Based on a pre-trained bucket extraction model, the target image is extracted to obtain the first complete bucket model from the target image perspective.

[0065] This application provides a cone-shaped barrel detection device that extracts a cone-shaped barrel model using a pre-trained cone-shaped barrel extraction model, thereby obtaining a first complete cone-shaped barrel model from the perspective of the target image. This improves the accuracy of cone-shaped barrel model establishment and lays the foundation for the accurate establishment of a cone-shaped barrel recognition model. Based on the first complete cone-shaped barrel model, a cone-shaped barrel recognition model is obtained, further improving the accuracy of cone-shaped barrel recognition model establishment.

[0066] Optionally, the cone-shaped barrel recognition model includes a segmenter and a discriminator.

[0067] The segmenter is used to extract multiple candidate images of cones from the image to be identified, and the discriminator is used to identify the real image containing the cone from the multiple candidate images of cones and the fake images of cones.

[0068] This application provides a cone-shaped barrel detection device. The cone-shaped barrel recognition model integrates a complete cone-shaped barrel model from multiple perspectives. Through a classifier and discriminator, it accurately identifies whether the image to be identified contains a cone-shaped barrel. Even occluded cone-shaped barrels can be accurately identified, thus improving the accuracy of cone-shaped barrel detection.

[0069] Alternatively, the position of the conical barrel can be located using the following method:

[0070] After determining that the image to be identified contains a cone-shaped barrel, the depth information and shooting position information corresponding to the image to be identified are obtained;

[0071] Obtain the depth information corresponding to the cone-shaped barrel from the depth information corresponding to the image to be identified;

[0072] Based on the depth information corresponding to the cone-shaped barrel and the shooting position information, the position information of the cone-shaped barrel is determined.

[0073] This application provides a device for detecting conical barrels, which, based on the depth information and shooting position information of the conical barrel, can quickly and accurately obtain the specific position information of the conical barrel, thereby improving the accuracy and speed of conical barrel detection.

[0074] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one executable instruction is stored therein, the executable instruction causing a processor to perform an operation corresponding to the above-described cone-shaped barrel detection method.

[0075] According to another aspect of this application, a computer device is provided, comprising: at least one processor coupled to a memory;

[0076] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described cone-shaped barrel detection method.

[0077] By employing the above-described technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

[0078] This application provides a method, apparatus, device, and medium for detecting cone-shaped barrels. Based on color images and depth information of cone-shaped barrels captured from multiple perspectives, it obtains cone-shaped barrel fusion features, identifies the image to be completed containing occluded cone-shaped barrels in the color images, and the target image of the unoccluded cone-shaped barrels. Based on the image to be completed, the cone-shaped barrel fusion features, and the target image, it obtains a cone-shaped barrel recognition model. The cone-shaped barrel recognition model accurately identifies cone-shaped barrels and determines their positions. Occluded cone-shaped barrels can also be accurately identified, greatly improving the accuracy of cone-shaped barrel detection.

[0079] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0080] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0081] Figure 1 A flowchart of a method for detecting a conical barrel according to an embodiment of this application is shown;

[0082] Figure 2 A flowchart of another method for detecting a conical barrel provided in an embodiment of this application is shown;

[0083] Figure 3 A flowchart of another method for detecting a conical barrel provided in an embodiment of this application is shown;

[0084] Figure 4 A flowchart of another method for detecting a conical barrel provided in an embodiment of this application is shown;

[0085] Figure 5 A flowchart of another method for detecting a conical barrel provided in an embodiment of this application is shown;

[0086] Figure 6 This diagram illustrates a block diagram of a detection device for a conical barrel according to an embodiment of this application.

[0087] in,

[0088] Figure 6 In the middle: 602-Feature fusion module; 604-First recognition module; 606-First acquisition module; 608-Second acquisition module; 610-Second recognition module. Detailed Implementation

[0089] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0090] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the specific embodiments, structures, features, and effects according to the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "an embodiment" or "an embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0091] To address the issue of low accuracy in current conical barrel testing methods, this application provides a method for detecting conical barrels, such as... Figure 1 As shown, the method includes:

[0092] 102: Based on the color images and depth information of the conical barrel taken from multiple perspectives, the three-dimensional features of the conical barrel corresponding to multiple perspectives are extracted, and the three-dimensional features of the conical barrel are fused to obtain the fused features of the conical barrel;

[0093] 104: Based on depth information, identify the image to be completed in a color image containing an occluded cone and the target image where the cone is not occluded;

[0094] 106: Extract the first complete cone-shaped barrel model from the target image viewpoint corresponding to the target image, and establish an initial cone-shaped barrel model based on the occluded cone-shaped barrels in the image to be completed, and use the cone-shaped barrel fusion features to complete the initial cone-shaped barrel model to obtain the second complete cone-shaped barrel model from the viewpoint of the image to be completed.

[0095] 108: Based on the first and second complete cone-shaped barrel models from multiple perspectives, a cone-shaped barrel recognition model is established;

[0096] 110: When an image to be identified is received, the cone-shaped barrel recognition model is used to identify whether the image contains a cone-shaped barrel and to locate the position of the cone-shaped barrel.

[0097] Specifically, dual cameras or a depth camera are used to acquire initial color images and depth information of the scene, which are then preprocessed. Next, 3D feature extraction is performed on the preprocessed color images and depth data. 3D features better reflect important characteristics of the bucket shape, including color, texture, shape, and spatial location. Then, the bucket features acquired from different viewpoints are fused to obtain a unified bucket feature. The shape of the bucket in the 3D image may change significantly when viewed from different perspectives. Through multi-view fusion, a unified and stable bucket feature is obtained. Objects within a color image are identified. Based on the object's depth information, a complete image containing an occluded bucket cone and an unoccluded target image of the bucket cone are obtained. A complete bucket cone model is extracted from the target image at its corresponding viewpoint. An initial bucket cone model is built for the occluded bucket cone based on the image to be completed. The initial bucket cone model is then completed using bucket cone fusion features to obtain the completed image. A second complete bucket cone model at the viewpoint of the image to be completed is then obtained based on the completed image. Based on the first and second complete bucket cone models at multiple different viewpoints, a bucket cone recognition model is obtained. An image to be recognized is acquired and input into the bucket cone recognition model. The model can identify whether the image contains a bucket cone; if so, its location can be accurately pinpointed.

[0098] This application provides a method for detecting cone-shaped barrels. Compared with existing technologies, this method obtains the three-dimensional features of the cone-shaped barrel corresponding to each viewpoint by acquiring color images and depth information of the cone-shaped barrel from multiple perspectives. The three-dimensional features of the cone-shaped barrel corresponding to each viewpoint are fused to obtain a fused cone-shaped barrel feature. This method identifies the image to be completed containing an occluded cone-shaped barrel in each color image, as well as the target image where the cone-shaped barrel is not occluded. A first complete cone-shaped barrel model is obtained from the target image's viewpoint. The fused cone-shaped barrel feature is used to complete the image to be completed containing the occluded cone-shaped barrel. Based on the completed image, a second complete cone-shaped barrel model is obtained from the viewpoint of the image to be completed. Based on the first and second complete cone-shaped barrel models from multiple perspectives, a cone-shaped barrel recognition model is established. The image to be recognized is then acquired, and the cone-shaped barrel recognition model is used to identify the image to determine whether it contains a cone-shaped barrel. If it does contain a cone-shaped barrel, its position information is determined. This method can recognize cone-shaped barrel images from multiple perspectives and accurately identify occluded cone-shaped barrels, thus improving the accuracy of cone-shaped barrel detection.

[0099] In one embodiment, data preprocessing of the initial color image and depth information includes preprocessing operations such as normalization, noise filtering, appropriate cropping and scaling, cleaning and organizing the collected data to prepare for subsequent feature extraction and analysis.

[0100] Specifically, a data preprocessing model is used for preprocessing. The input data consists of the original color image and its corresponding depth information, and the output data consists of the preprocessed color image and depth information. Preprocessing includes normalization, noise filtering, and appropriate cropping and scaling. Normalization of the numerical range of the color image and depth information typically involves scaling pixel values ​​to the range of [0, 1] or [-1, 1]. This avoids the influence of numerical units and ranges on model training. Median filters or Gaussian filters are used to remove noise from the image to obtain a clearer image. Appropriate cropping and scaling reduce computational load and improve processing speed. After these preprocessing operations, the processed color image and depth information are obtained, which can be directly used in the subsequent 3D feature extraction stage.

[0101] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another method for detecting conical barrels is provided, such as... Figure 2 As shown, the method includes:

[0102] Step 201: Merge the color image and depth information corresponding to each viewpoint to obtain the comprehensive image corresponding to each viewpoint. Based on the pre-trained 3D feature extraction model, perform 3D feature extraction on each comprehensive image to obtain the cone-shaped barrel 3D features corresponding to each viewpoint.

[0103] Step 202: Based on the shooting position information corresponding to each viewpoint and the position information corresponding to the reference viewpoint, calculate the viewpoint transformation matrix corresponding to each viewpoint when it is converted into the reference viewpoint;

[0104] Step 203: Based on the viewpoint transformation matrix and the three-dimensional features of the cone barrel corresponding to each viewpoint, obtain the three-dimensional transformation features of the cone barrel corresponding to each viewpoint when it is converted into a reference viewpoint;

[0105] Step 204: Fuse the 3D transformation features of the cone-shaped barrel corresponding to each viewpoint to obtain the cone-shaped barrel fused features.

[0106] Specifically, the pre-trained 3D feature extraction model is a 3D convolutional neural network (3D-CNN) model. The pre-processed color image and depth information are input into the pre-trained 3D feature extraction model, and the model's output is the extracted 3D features. The 3D convolutional neural network model can acquire the stereoscopic information of an image, thereby extracting richer and more representative features, such as color, texture, shape, and spatial location.

[0107] In one specific implementation, the preprocessed color image and depth information corresponding to each viewpoint are merged into a three-channel composite image, forming a new three-dimensional tensor. This composite image serves as the input to a three-dimensional convolutional neural network model, and the output of the three-dimensional convolutional neural network model is the three-dimensional feature of the cone-shaped barrel corresponding to that viewpoint.

[0108] The three-dimensional convolutional neural network model performs the following steps:

[0109] 1. Convolution: A convolutional layer can be viewed as a filtering operation on the input data. Each convolutional kernel slides in space and performs a dot product operation with the input data, generating a new feature map. In 3D convolutional neural network models, convolution also considers depth information, enabling the model to capture information such as the shape and position of objects in 3D space.

[0110] 2. Linear Activation: Activation functions introduce non-linearity, enabling the model to fit more complex patterns. Commonly used activation functions include ReLU, sigmoid, and tanh.

[0111] 3. Pooling: Pooling layers are used to reduce the dimensionality of feature maps, reduce computation, and retain important features while enhancing the translation invariance of the model.

[0112] 4. Fully connected: The fully connected layer transforms the output (feature map) of the previous layer into a one-dimensional vector and obtains a new feature vector through matrix transformation.

[0113] 5. Feature Extraction: In a 3D convolutional neural network model, the output of a certain layer (usually a fully connected layer or the last convolutional layer) is typically selected as the feature representation of the input data. These operations are performed sequentially, forming the forward propagation process.

[0114] Specifically, the epipolar geometry principle is used to fuse the cone-shaped barrel features acquired from different viewpoints. The input is the cone-shaped barrel features acquired from multiple viewpoints, and the output is the fused, unified, and stable cone-shaped barrel features. Specifically, some form of geometric transformation (such as affine or perspective transformation) can be used to map the features of all viewpoints onto a common reference frame (reference viewpoint). This requires adjusting the original features through some geometric transformation, such as affine or perspective transformation, to match them with the reference frame. Based on the shooting position information corresponding to each viewpoint and the position information corresponding to the reference viewpoint, the viewpoint transformation matrix corresponding to each viewpoint when transformed into the reference viewpoint is calculated.

[0115] After obtaining the geometric transformation matrices from each viewpoint to the reference frame, these matrices are used to transform the features of each viewpoint to obtain the feature representation under the reference frame. Specifically, the viewpoint transformation matrix corresponding to each viewpoint is multiplied with the cone-shaped barrel 3D feature to obtain the cone-shaped barrel 3D transformed feature corresponding to the reference viewpoint.

[0116] Feature fusion combines features from multiple perspectives into a unified feature representation. There are various possible feature fusion strategies, one common one being voting. The concept of "voting" originates from ensemble learning, whose core idea is to have models predict data, then vote on the predictions from each model, selecting the result with the most votes as the final result. In the context of multi-view fusion, voting is mainly reflected in the aggregation process of features from different perspectives. Specifically, for each feature dimension, feature values ​​from different perspectives can "vote" to determine the final value of that dimension. Common voting strategies include taking the mean (average voting) and taking the mode (majority rule). For example, suppose there are three perspectives, each providing a feature vector of length 10. Then, for the first element of the final feature vector, the first elements of the feature vectors from the three perspectives can vote: if average voting is used, the average of these three elements is taken; if mode voting is used, the value that appears most frequently among the three elements is taken. In this way, a unified feature representation that integrates information from all perspectives can be obtained, which is very useful for subsequent steps such as occlusion processing, detection, and localization.

[0117] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another method for detecting conical barrels is provided, such as... Figure 3 As shown, the method includes:

[0118] Step 301: Perform object recognition on the color image, obtain the image of each object contained in the color image, and determine the front-to-back relationship between objects in the color image based on the depth information corresponding to the color image.

[0119] Step 302: Based on the front-back relationship between objects in the color image, the image of the object and the preset cone image, determine the image to be completed for the occluded cone and the target image of the unoccluded cone.

[0120] Step 303: Based on the pre-trained bucket extraction model, the bucket model is extracted from the target image to obtain the first complete bucket model from the target image perspective.

[0121] Step 304: Establish an initial cone model based on the occluded cones in the image to be completed, and generate a cone prediction image based on the initial cone model;

[0122] Step 305: Adjust the predicted cone-shaped image using the cone-shaped barrel fusion feature to obtain the completed cone-shaped barrel image;

[0123] Step 306: Based on the pre-trained bucket extraction model, extract the bucket model from the completed bucket image to obtain the second complete bucket model from the perspective of the image to be completed.

[0124] Specifically, object recognition in color images relies on the depth information of the identified objects within the color image. In images captured using a depth camera, each pixel value represents the distance from the corresponding spatial point to the camera. Therefore, these depth values ​​can be directly used to determine the spatial relationships between objects: objects with smaller depth values ​​are located in front of objects with larger depth values.

[0125] For an occluded bucket, an initial bucket model is built using the features and shape information of the visible portion, along with the standard models of other unoccluded buckets of the same category. This creates a preliminary, and perhaps not entirely accurate, representation of the occluded object (bucket). This representation, or "model," reflects the object's basic features and shape, providing a starting point for subsequent iterative optimization. This "model" can take various forms, depending on the nature of the problem and the method used. For example, it can be a two-dimensional contour map or a three-dimensional voxel model; it can include attributes such as the object's color and texture, or only its shape information. Furthermore, the construction of this initial bucket model typically utilizes known prior information, such as the average model of objects in the same category or the object's symmetry. For instance, if the object is known to be symmetrical, even if only half of the object is seen, it can be assumed that the other half is symmetrical to the visible portion, thus obtaining a complete initial bucket model.

[0126] Based on the initial model, a "guess-and-verify" strategy is adopted to gradually recover the occluded parts through iteration. Specifically, firstly, a predicted bucket image is generated based on the initial model and known conditions such as viewpoint and lighting. Then, this predicted image is compared with the bucket fusion features, and the model is adjusted according to the differences. Alternatively, it can be compared with the actual image. This process is repeated until a satisfactory result is achieved. Shape matching and image registration are commonly used computer vision techniques that can be used to measure and adjust the similarity between two shapes or images. When dealing with occlusion problems, these techniques can be used to fine-tune the initial model to make it closer to the actual object shape. For example, if the color of a certain region in the predicted image does not match the bucket fusion features, the color of the corresponding region in the predicted image needs to be adjusted.

[0127] In one embodiment, a cone-shaped barrel recognition model is established based on a first complete cone-shaped barrel model and a second complete cone-shaped barrel model from multiple perspectives. The cone-shaped barrel recognition model is then optimized and adjusted. Alternatively, when using the cone-shaped barrel recognition model to detect cone-shaped barrels in the image to be recognized, the cone-shaped barrel recognition model is optimized and adjusted based on the detection results.

[0128] Based on real-time feedback data of the detection results of the images to be recognized, the parameters of the bucket cone recognition model are continuously optimized and adjusted to improve the robustness and accuracy of the algorithm. The input is the system's operating status and feedback information, and the output is the updated and optimized bucket cone recognition model. Specifically, online learning or incremental learning methods can be used to dynamically adjust the bucket cone recognition model parameters based on real-time feedback information. For example, if a certain type of bucket is frequently misdetected, samples of these buckets can be collected for further training and optimization of the model. In the real-time feedback and optimization phase, the system will dynamically adjust and optimize the bucket cone recognition model based on the operating status and feedback information. Therefore, this phase may involve the following steps, such as... Figure 4 As shown:

[0129] 1. Monitoring and Data Collection: First, it is necessary to continuously monitor the system's operational status and collect relevant data and feedback information. This may include model outputs, user feedback, system performance metrics, etc.

[0130] 2. Analyze and understand the data: After obtaining the data, it is necessary to analyze and understand it to identify the problems and shortcomings of the model, as well as possible directions for improvement.

[0131] 3. Update the model: Based on the above analysis results, update the model. This may involve modifying the model's parameters or retraining the model using new training data.

[0132] 4. Test the new model: After updating the model, you need to evaluate the performance of the new model on the test set to ensure that it achieves the expected results.

[0133] 5. Deploy the new model: If the performance of the new model meets the requirements, it will be deployed to the production environment to replace the old model.

[0134] In one embodiment, the barrel conicity identification model includes a segmenter and a discriminator.

[0135] The segmenter is used to extract multiple bucket cone candidate images from the image to be identified, and the discriminator is used to identify the real image containing the bucket cone from the multiple bucket cone candidate images and fake bucket cone images.

[0136] In one embodiment, the bucket identification model is based on generative adversarial networks (GANs) for image segmentation and a classifier for identification. The input is the image to be identified, and the output is the precise location of the bucket and its category label (i.e., whether it is a bucket). Specifically, the GAN can accurately segment the bucket from the background, while the classifier confirms whether a detected object is a bucket. During the detection and localization phases, the GAN can be used for image segmentation. This allows it to map from the original image to the segmented image.

[0137] The general process for generating adversarial networks is as follows:

[0138] 1. Generator Training: The generator uses random noise as input and produces fake images. These fake images, along with real images, are labeled as positive (real images) and negative (fake images) and fed into a discriminator for discrimination. The generator's parameters are updated under an adversarial loss function to generate increasingly realistic fake images.

[0139] 2. Discriminator Training: With the generator parameters fixed, the discriminator is trained on real samples. Its goal is to correctly distinguish between real and fake images. These steps are repeated alternately until the model converges, yielding the trained model.

[0140] In one embodiment, after determining that the image to be identified contains a cone, the location information of the cone is identified using the following method:

[0141] After determining that the image to be identified contains a cone-shaped barrel, obtain the depth information and shooting location information of the image to be identified;

[0142] Obtain the depth information corresponding to the cone-shaped barrel from the depth information corresponding to the image to be identified;

[0143] Based on the depth information and shooting position information corresponding to the cone, the position information of the cone is determined.

[0144] Specifically, the depth information of the cone to be identified is obtained from the depth information corresponding to the image to be identified, and the specific location information of the cone is obtained based on the depth information of the cone and the shooting position information.

[0145] In one specific implementation, such as Figure 5 As shown, the following steps are used to identify cone-shaped barrels:

[0146] 1. Data Preprocessing Stage

[0147] Input: Original color image and depth information of the occluded object

[0148] Output: Preprocessed color image and depth information

[0149] Steps include data cleaning (e.g., removing noise and irrelevant information), format conversion (e.g., resizing images to a uniform size and color space), and normalization.

[0150] 2. Three-dimensional feature extraction stage

[0151] Input: Preprocessed color image and depth information

[0152] Output: 3D features

[0153] Steps: Based on the preprocessed color image and depth information, extract useful 3D features, such as shape, texture, and color.

[0154] 3. Multi-perspective fusion stage

[0155] Input: 3D features

[0156] Output: Fusion result

[0157] Steps: Fuse 3D features from different perspectives to obtain a more complete and accurate representation of the object.

[0158] 4. Obstruction Treatment Stage

[0159] Input: Fusion result

[0160] Output: Initial model

[0161] Steps: Use the depth information in the fused result to determine the relationship between objects and sort them; build an initial model for objects with occlusion relationships; and restore the occluded parts through iterative optimization.

[0162] 5. Detection and Positioning Stage:

[0163] Input: Image to be recognized

[0164] Output: Segmentation results generated by the adversarial network.

[0165] Steps: Use generative adversarial networks for image segmentation.

[0166] 6. Real-time feedback and optimization phase:

[0167] Input: Generate adversarial network segmentation results

[0168] Output: Optimized model

[0169] Steps: Monitor and collect system operating status and feedback information, analyze and understand the collected data, update the model based on the analysis results, test the new model, and if the test is satisfactory, deploy the new model; otherwise, continue to analyze the data and update the model.

[0170] Furthermore, as a response to the above Figure 1 The implementation of the method shown in this embodiment of the invention provides a detection device for a conical barrel, such as... Figure 6 As shown, the device includes:

[0171] The feature fusion module 602 is used to extract the three-dimensional features of the cone-shaped barrel corresponding to multiple viewpoints based on the color images and depth information of the cone-shaped barrel taken from multiple viewpoints, and to fuse the three-dimensional features of the cone-shaped barrel to obtain the fused features of the cone-shaped barrel.

[0172] The first recognition module 604 is used to recognize, based on depth information, a color image containing an image to be completed with an occluded cone and a target image with an unoccluded cone.

[0173] The first acquisition module 606 is used to extract the first complete cone barrel model under the view of the target image corresponding to the target image, and to establish an initial cone barrel model based on the occluded cone barrel in the image to be completed, and to complete the initial cone barrel model using the cone barrel fusion feature to obtain the second complete cone barrel model under the view of the image to be completed.

[0174] The second acquisition module 608 is used to establish a cone-shaped barrel recognition model based on the first complete cone-shaped barrel model and the second complete cone-shaped barrel model under multiple perspectives.

[0175] The second recognition module 610 is used to identify whether the image to be recognized contains a cone and to locate the position of the cone when the image to be recognized is received, by using a cone recognition model.

[0176] This application provides a cone-shaped barrel detection device. Compared with the prior art, it obtains the three-dimensional features of the cone-shaped barrel corresponding to each viewpoint by acquiring color images and depth information of the cone-shaped barrel from multiple perspectives. The three-dimensional features of the cone-shaped barrel corresponding to each viewpoint are fused to obtain the cone-shaped barrel fusion feature. It identifies the image to be completed containing the occluded cone-shaped barrel in each color image and the target image where the cone-shaped barrel is not occluded. It obtains the first complete cone-shaped barrel model from the viewpoint of the target image. The cone-shaped barrel fusion feature is used to complete the image to be completed containing the occluded cone-shaped barrel to obtain the second complete cone-shaped barrel model from the viewpoint of the image to be completed. Based on the first and second complete cone-shaped barrel models from multiple perspectives, a cone-shaped barrel recognition model is established. When an image to be identified is received, the cone-shaped barrel recognition model is used to identify whether the image contains a cone-shaped barrel. If it contains a cone-shaped barrel, the position of the cone-shaped barrel is located. It can identify cone-shaped barrel images from multiple perspectives and accurately identify occluded cone-shaped barrels, thus improving the accuracy of cone-shaped barrel detection.

[0177] In one embodiment, the feature fusion module is further used for:

[0178] The color image and depth information corresponding to each viewpoint are merged to obtain the comprehensive image corresponding to each viewpoint. Based on the pre-trained 3D feature extraction model, 3D features are extracted from each comprehensive image to obtain the 3D features of the cone barrel corresponding to each viewpoint.

[0179] Based on the shooting position information corresponding to each viewpoint and the position information corresponding to the reference viewpoint, calculate the viewpoint transformation matrix corresponding to each viewpoint when it is transformed into the reference viewpoint;

[0180] Based on the viewpoint transformation matrix and the 3D features of the cone barrel corresponding to each viewpoint, the 3D transformation features of the cone barrel corresponding to each viewpoint are obtained when each viewpoint is transformed into a reference viewpoint.

[0181] The 3D transformation features of the cone-shaped barrel corresponding to each viewpoint are fused to obtain the cone-shaped barrel fused features.

[0182] In one embodiment, the first identification module is used for:

[0183] Perform object recognition on a color image to obtain an image of each object contained in the color image;

[0184] Based on the depth information corresponding to the color image, determine the front-to-back relationship between objects in the color image;

[0185] Based on the front-to-back relationship between objects in the color image, the images of the objects, and the preset cone image, the image to be completed for the occluded cone and the target image for the unoccluded cone are determined.

[0186] In one embodiment, the first acquisition module is further configured to:

[0187] Based on the initial cone-shaped barrel model, generate a predicted cone-shaped barrel image;

[0188] The predicted image of the cone-shaped barrel is adjusted using the cone-shaped barrel fusion feature to obtain the completed image of the cone-shaped barrel;

[0189] Based on the completed cone-shaped barrel image, a second complete cone-shaped barrel model is extracted from the perspective of the image to be completed.

[0190] In one embodiment, the first acquisition module is further configured to:

[0191] Based on a pre-trained bucket extraction model, a bucket model is extracted from the target image to obtain the first complete bucket model from the target image's perspective.

[0192] In one embodiment, the barrel conicity identification model includes a segmenter and a discriminator.

[0193] The segmenter is used to extract multiple bucket cone candidate images from the image to be identified, and the discriminator is used to identify the real image containing the bucket cone from the multiple bucket cone candidate images and fake bucket cone images.

[0194] In one embodiment, the position of the conical barrel is located using the following method:

[0195] After determining that the image to be identified contains a cone-shaped barrel, obtain the depth information and shooting location information of the image to be identified;

[0196] Obtain the depth information corresponding to the cone-shaped barrel from the depth information corresponding to the image to be identified;

[0197] Based on the depth information and shooting position information corresponding to the cone, the position information of the cone is determined.

[0198] According to one embodiment of the present invention, a computer-readable storage medium is provided, which stores at least one executable instruction that can perform the operation corresponding to the above-described cone barrel detection method.

[0199] According to one embodiment of the present invention, a computer device is provided, including at least one processor coupled to a memory;

[0200] The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the above-described cone-shaped barrel detection method.

[0201] It will be apparent to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. In one embodiment, they can be implemented using device-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0202] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A method for detecting a conical barrel, characterized in that, include: Based on the color images and depth information of the conical barrel taken from multiple perspectives, the three-dimensional features of the conical barrel corresponding to multiple perspectives are extracted, and the three-dimensional features of the conical barrel are fused to obtain the fused features of the conical barrel. Based on the depth information, the color image is identified to include a complete image of an occluded cone and a target image of an unoccluded cone. Extract the first complete cone-shaped barrel model from the target image viewpoint corresponding to the target image, and establish an initial cone-shaped barrel model based on the occluded cone-shaped barrels in the image to be completed, and use the cone-shaped barrel fusion feature to complete the initial cone-shaped barrel model to obtain the second complete cone-shaped barrel model from the viewpoint of the image to be completed. A cone-shaped barrel recognition model is established based on the first and second complete cone-shaped barrel models from multiple perspectives. When an image to be identified is received, the cone-shaped barrel recognition model is used to identify whether the image contains a cone-shaped barrel and to locate the position of the cone-shaped barrel.

2. The method for detecting conical barrels as described in claim 1, characterized in that, The method involves extracting three-dimensional features of the conical barrel from multiple perspectives based on color images and depth information captured from various viewpoints, and then fusing these three-dimensional features to obtain fused features of the conical barrel, including: The color image and depth information corresponding to each viewpoint are merged to obtain the comprehensive image corresponding to each viewpoint. Based on the pre-trained 3D feature extraction model, 3D features are extracted from each comprehensive image to obtain the 3D features of the cone barrel corresponding to each viewpoint. Based on the shooting position information corresponding to each viewpoint and the position information corresponding to the reference viewpoint, calculate the viewpoint transformation matrix corresponding to each viewpoint when it is transformed into the reference viewpoint; Based on the viewpoint transformation matrix and the three-dimensional features of the cone barrel corresponding to each viewpoint, the three-dimensional transformation features of the cone barrel corresponding to each viewpoint are obtained when each viewpoint is transformed into a reference viewpoint. The 3D transformation features of the cone-shaped barrel corresponding to each viewpoint are fused to obtain the cone-shaped barrel fused features.

3. The method for detecting conical barrels as described in claim 1, characterized in that, The step of identifying, based on the depth information, the color image containing the image to be completed with the occluded cone and the target image with the unoccluded cone includes: Perform object recognition on the color image to obtain an image of each object contained in the color image; Based on the depth information corresponding to the color image, the front-to-back relationship between objects in the color image is determined; Based on the front-to-back relationship between objects in the color image, the images of the objects, and the preset cone image, the image to be completed for the occluded cone and the target image for the unoccluded cone are determined.

4. The method for detecting conical barrels as described in claim 1, characterized in that, The process of using the cone-shaped barrel fusion features to complete the initial cone-shaped barrel model to obtain a second complete cone-shaped barrel model from the perspective of the image to be completed includes: Based on the initial cone-shaped barrel model, a predicted cone-shaped barrel image is generated; The cone-shaped barrel fusion feature is used to adjust the predicted cone-shaped barrel image to obtain the completed cone-shaped barrel image; Based on the completed cone-shaped barrel image, a second complete cone-shaped barrel model is extracted from the perspective of the image to be completed.

5. The method for detecting conical barrels as described in claim 1, characterized in that, The extraction of the first complete cone-shaped barrel model from the target image perspective corresponding to the target image includes: Based on a pre-trained bucket extraction model, the target image is extracted to obtain the first complete bucket model from the target image perspective.

6. The method for detecting conical barrels as described in claim 1, characterized in that, The cone-shaped barrel recognition model includes a segmenter and a discriminator. The segmenter is used to extract multiple candidate images of cones from the image to be identified, and the discriminator is used to identify the real image containing the cone from the multiple candidate images of cones and the fake images of cones.

7. The method for detecting conical barrels as described in claim 5, characterized in that, The position of the conical barrel is located using the following method: After determining that the image to be identified contains a cone-shaped barrel, the depth information and shooting position information corresponding to the image to be identified are obtained; Obtain the depth information corresponding to the cone-shaped barrel from the depth information corresponding to the image to be identified; Based on the depth information corresponding to the cone-shaped barrel and the shooting position information, the position information of the cone-shaped barrel is determined.

8. A detection device for a conical barrel, characterized in that, include: The feature fusion module is used to extract the three-dimensional features of the cone-shaped barrel corresponding to multiple viewpoints based on the color images and depth information of the cone-shaped barrel taken from multiple viewpoints, and to fuse the three-dimensional features of the cone-shaped barrel to obtain the fused features of the cone-shaped barrel. The first recognition module is used to identify, based on the depth information, the color image containing the image to be completed with the occluded cone and the target image with the unoccluded cone; The first acquisition module is used to extract the first complete cone-shaped barrel model from the target image viewpoint corresponding to the target image, and to establish an initial cone-shaped barrel model based on the occluded cone-shaped barrels in the image to be completed, and to complete the initial cone-shaped barrel model using the cone-shaped barrel fusion feature to obtain the second complete cone-shaped barrel model from the viewpoint of the image to be completed. The second acquisition module is used to establish a cone-shaped barrel recognition model based on the first complete cone-shaped barrel model and the second complete cone-shaped barrel model from multiple perspectives. The second recognition module is used to identify whether the image to be recognized contains a cone and to locate the position of the cone when the image to be recognized is received, by means of the cone recognition model.

9. A computer-readable storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the detection method for a conical barrel as described in any one of claims 1-7.

10. A computer device comprising at least one processor coupled to a memory; The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the detection method of the conical barrel as described in any one of claims 1-7.

Images