Image recognition method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image recognition method and device, equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: carrying out the format alignment of a two-dimensional image and a three-dimensional image of an obtained operation region, and obtaining corresponding two-dimensional data and three-dimensional data; performing feature extraction on the two-dimensional data and the three-dimensional data to obtain extraction results, and fusing the extraction results based on the dynamic attention weight to obtain a fused feature map; performing target identification on the fused feature map to obtain a target object in the fused feature map; and extracting local features of the target object, and classifying the target object by using a classifier according to the local features to obtain a classification result. According to the garbage cleaning equipment, efficient recognition and classification of various kinds of garbage in the operation process of the garbage cleaning equipment are achieved, and the operation efficiency of the garbage cleaning equipment is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image recognition method, apparatus, device, and storage medium. Background Technology

[0002] With the acceleration of urbanization and the continuous increase in waste production, people have increasingly higher demands for the intelligence of waste disposal equipment such as robotic vacuum cleaners and intelligent sweeping vehicles. These devices not only need waste cleaning functions, but also waste recognition functions to better classify and clean up waste. However, traditional waste sorting methods are inefficient and cannot meet the ever-growing demand for waste disposal.

[0003] In the patrol scenarios of robotic vacuum cleaners and intelligent cleaning vehicles, garbage identification faces a series of problems: the shape of garbage is often irregular, such as long strips or curved piles, which makes accurate identification and location marking difficult; the patrol environment may also have problems such as changes in lighting and occlusion, which also increases the difficulty of garbage identification; affected by the complex environment, traditional visual algorithms face the problem of missing small targets in complex backgrounds in urban street garbage detection, such as cigarette butts and paper scraps, which are easily overlooked; when garbage is damaged, dirty, or deformed, ordinary image recognition technology often fails to accurately identify the type of garbage. One or more of the above problems make the current garbage identification and classification inefficient. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an image recognition method, apparatus, device and storage medium to solve the problem of low efficiency in existing waste identification and classification methods.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, the present invention provides an image recognition method, comprising: The format of the acquired 2D and 3D images of the work area is aligned to obtain the corresponding 2D and 3D data. Feature extraction is performed on the two-dimensional data and the three-dimensional data to obtain extraction results, and the extraction results are fused based on dynamic attention weights to obtain a fused feature map; Target recognition is performed on the fused feature map to obtain the target object in the fused feature map; Local features of the target object are extracted, and a classifier is used to classify the target object based on the local features to obtain a classification result.

[0006] In an optional implementation, the two-dimensional data includes image data and infrared data, and the extraction result includes an image feature map, an infrared feature map, and a three-dimensional feature map. The step of performing feature extraction on the two-dimensional data and the three-dimensional data to obtain the extraction result includes: The image data is used to extract features using five convolutional layers and four pooling layers to obtain two-dimensional features. The two-dimensional features are then projected onto a fusion network using a viewpoint transformation matrix to obtain the image feature map in a preset format. Infrared features are obtained by using depthwise separable convolutions of a lightweight network to extract features from the infrared data, and the infrared features are projected onto the fusion network to obtain the infrared feature map in a preset format. The three-dimensional data is mapped onto the fusion network to obtain a sparse feature map. The sparse feature map is then processed by three sparse convolutional layers and two pooling layers to extract three-dimensional features. These three-dimensional features are then converted into a three-dimensional feature map in the preset format.

[0007] In an optional implementation, the step of fusing the extracted results based on dynamic attention weights to obtain a fused feature map includes: A dynamic attention weight calculation model is constructed, and the lighting conditions for acquiring the two-dimensional image and the three-dimensional image are determined. The image feature map, the infrared feature map, the three-dimensional feature map, and the illumination conditions are input into the dynamic attention weight calculation model to obtain the first weight corresponding to the image feature map, the second weight corresponding to the infrared feature map, and the third weight corresponding to the three-dimensional feature map; The image feature map, the infrared feature map, and the three-dimensional feature map are fused according to the first weight, the second weight, and the third weight to obtain the fused feature map.

[0008] In an optional implementation, the step of performing target recognition on the fused feature map to obtain the target object in the fused feature map includes: Obtain the type of the target object, construct a corresponding detection head based on the type, and each type of target object corresponds to a detection branch in a detection head; The fused feature map is input into the detection head, and the recognition result of the fused feature map is output through the detection branch of the detection head.

[0009] In an optional implementation, after the step of obtaining the target object in the fused feature map, the method further includes: If the target object is a small target object and its local features cannot be extracted, then the two-dimensional image corresponding to the small target object is obtained; The two-dimensional image is annotated to obtain the image coordinates corresponding to the small target object, and the image coordinates are mapped to a preset fusion space; The distribution density of the small target object in the preset fusion space is obtained, and the resolution of the fusion space is adjusted according to the distribution density to obtain the small target feature corresponding to the small target object in the preset fusion space; The small target features are enhanced to obtain enhanced features. The enhanced features are then fused with the corresponding three-dimensional features of the three-dimensional image to obtain a fusion result. The fusion result is used as the recognition result of the small target object.

[0010] In an optional implementation, the step of fusing the enhanced features with the three-dimensional features corresponding to the three-dimensional image to obtain a fusion result includes: Obtain the confidence level corresponding to the small target feature, and fill the small target feature with a confidence level greater than the confidence threshold into the three-dimensional feature to obtain the fused three-dimensional feature; The enhanced features and the fused 3D features are weighted and fused to obtain the fusion result.

[0011] In an optional implementation, the small target features include texture features and semantic features, and the step of performing feature enhancement on the small target features to obtain enhanced features includes: High-frequency texture features are extracted from the texture features, and channel-weighted and spatial-weighted high-frequency texture features are applied to obtain optimized high-frequency texture features. Low-frequency texture features are extracted from the texture features, and low-pass filtering and multi-scale low-frequency fusion are performed on the low-frequency texture features to obtain optimized low-frequency texture features. The optimized high-frequency texture features and the optimized high-frequency texture features are then fused in the frequency domain to obtain enhanced texture features. High-level semantic features and low-level detail features are extracted from the semantic features, and the high-level semantic features and low-level detail features are fused together through a feature pyramid to obtain enhanced semantic features; The enhanced features are obtained by weighted fusion of the enhanced texture features and the enhanced semantic features.

[0012] In an optional implementation, the step of obtaining the two-dimensional image corresponding to the small target object includes: The small target detection model is used to extract targets from the two-dimensional image of the work area to obtain the two-dimensional image corresponding to the small target object; The training process of the small object detection model includes: A target detection model is constructed by inputting training images into the target detection model and constructing a weighted loss function to guide the training of the target detection model; the weighted loss function includes target classification loss, small target localization loss, and small target missed detection loss; The target detection model corresponding to the weighted loss function when the loss value is less than or equal to the preset loss value is taken as the small target detection model.

[0013] In a second aspect, the present invention provides an image recognition device, comprising: The image alignment module also needs to perform format alignment on the two-dimensional and three-dimensional images of the acquired work area to obtain the corresponding two-dimensional and three-dimensional data. The feature fusion module is used to extract features from the two-dimensional data and the three-dimensional data to obtain extraction results, and to fuse the extraction results based on dynamic attention weights to obtain a fused feature map; The target recognition module is used to perform target recognition on the fused feature map to obtain the target object in the fused feature map; The target classification module is used to extract local features of the target object and use a classifier to classify the target object based on the local features to obtain a classification result.

[0014] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the image recognition method described in the first aspect.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image recognition method described in the first aspect.

[0016] The present invention provides an image recognition method, apparatus, device, and storage medium that, by constructing an architecture for image acquisition, image processing, feature fusion, and target classification, achieves end-to-end processing of images in the work area, enabling efficient identification and classification of various types of waste by the waste cleaning equipment during operation, thereby improving the operational efficiency of the waste cleaning equipment.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention is shown; Figure 2 A flowchart illustrating an image recognition method provided by an embodiment of the present invention is shown; Figure 3 A schematic diagram illustrating the principle of an image recognition method provided by an embodiment of the present invention is shown; Figure 4 A flowchart illustrating a small target object recognition method provided by an embodiment of the present invention is shown; Figure 5 A functional block diagram of an image recognition device provided in an embodiment of the present invention is shown.

[0020] icon: 100 - Electronic device; 110 - Memory; 120 - Processor; 130 - Communication module; 500 - Image recognition device; 501 - Image alignment module; 502 - Feature fusion module; 503 - Target recognition module; 504 - Target classification module. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0024] Please refer to Figure 1 , Figure 1 This is a block diagram of an electronic device 100 provided in this embodiment. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0025] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0026] The processor 120 is used to read / write data or programs stored in the memory 110 and to perform corresponding functions.

[0027] The communication module 130 is used to establish a communication connection between the server and other communication terminals through the network, and to send and receive data through the network.

[0028] It should be understood that, Figure 1 The structure shown is only a schematic diagram of the electronic device 100. The electronic device 100 may also include components that are larger than... Figure 1The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0029] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating an image recognition method provided in this embodiment. The method includes: S201. Align the formats of the two-dimensional and three-dimensional images of the acquired work area to obtain the corresponding two-dimensional and three-dimensional data.

[0030] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the principle of an image recognition method provided in this embodiment.

[0031] During the operation of waste collection equipment, various factors such as environment, lighting, and angle can affect the accuracy and real-time performance of waste identification, leading to incomplete or untimely waste collection. Therefore, equipping the waste collection equipment with multiple image acquisition devices can enable comprehensive identification of waste in the work area. Commonly used image acquisition devices include RGB cameras, infrared sensors, and LiDAR. Cameras and infrared sensors are primarily used to acquire two-dimensional images, while LiDAR is used to acquire three-dimensional images.

[0032] For example, multiple image acquisition devices can be controlled to trigger synchronously, with the LiDAR outputting a frame of point cloud data every 100ms and the RGB camera and infrared sensor outputting a frame of image data every 33ms.

[0033] Since cameras typically capture images of the waste cleaning equipment's operating area at a certain angle, the operating area is usually a fan-shaped area. To prevent interference from other information, the area captured by the camera can be cropped to obtain the actual operating area.

[0034] Since the shooting angle and shooting time of two-dimensional and three-dimensional images may be different, after acquiring two-dimensional and three-dimensional images, it is necessary to align the formats of the two-dimensional and three-dimensional images from various aspects to obtain two-dimensional and three-dimensional data in a unified format.

[0035] S202. Feature extraction is performed on the two-dimensional data and the three-dimensional data to obtain extraction results. The extraction results are then fused based on dynamic attention weights to obtain a fused feature map.

[0036] After extracting features from two-dimensional and three-dimensional data, since the two-dimensional features corresponding to the two-dimensional data do not contain spatial features, the two-dimensional features and three-dimensional features cannot be directly fused. Therefore, it is necessary to map the feature extraction results of the two-dimensional and three-dimensional data to a unified fusion space for fusion to obtain a fused feature map.

[0037] S203. Perform target recognition on the fused feature map to obtain the target object in the fused feature map.

[0038] After obtaining the fused feature map, target recognition can be performed on the fused feature map to identify the target objects in the fused feature map. The target objects can be various types of garbage to be cleaned up, such as plastic bottles, aluminum cans, waste paper, packaging bags, etc.

[0039] S204. Extract the local features of the target object, and use a classifier to classify the target object according to the local features to obtain the classification result.

[0040] Since different types of waste may require sorting and disposal, after identifying the target object, it can be classified based on its local features to determine the type of waste, and then the corresponding disposal method can be selected based on the type of waste.

[0041] This embodiment constructs an architecture for image acquisition, image processing, feature fusion, and target classification, achieving end-to-end processing of images in the work area. This enables the waste cleaning equipment to efficiently identify and classify various types of waste during operation, thereby improving the operational efficiency of the waste cleaning equipment.

[0042] In one implementation, the step of format aligning the two-dimensional and three-dimensional images of the acquired work area includes: Determine the first timestamp corresponding to the two-dimensional image and the second timestamp corresponding to the three-dimensional image; Time compensation is performed on the first timestamp based on the second timestamp to align the time of the two-dimensional image with that of the three-dimensional image. The extrinsic parameters of the two-dimensional image acquisition device and the three-dimensional image acquisition device are calibrated, and the two-dimensional image and the three-dimensional image are transferred to the same coordinate system for spatial alignment using a visual feature matching algorithm.

[0043] Since the time intervals for acquiring images of the work area by RGB cameras, infrared sensors, and LiDAR may be inconsistent, resulting in different acquisition times for 2D and 3D images, corresponding timestamps can be assigned to the captured 2D and 3D images. Then, using the timestamp of one of the 2D and 3D images as a reference, time compensation is performed on the timestamp of the other, thereby achieving time alignment between the 2D and 3D images.

[0044] In addition, since the extrinsic parameters of RGB cameras, infrared sensors and LiDAR may be different, and the extrinsic parameters are a set of parameters that describe the position and attitude of the camera in the world coordinate system, it is necessary to calibrate the extrinsic parameters of RGB cameras, infrared sensors and LiDAR first to make the extrinsic parameters of each device the same.

[0045] Then, a recognition feature matching algorithm is used to transfer the two-dimensional and three-dimensional images to the same coordinate system, such as to BEV (Bird's Eye View), to achieve spatial alignment between the two-dimensional and three-dimensional images.

[0046] For example, the 3D image is converted into an N×4 matrix (N is the number of point clouds, and the four dimensions are the X, Y, Z coordinates and the reflection intensity, respectively). The RGB camera image is set to 640×640 resolution, normalized to the [0,1] interval, and converted to CHW format (3 channels, 640 height, 640 width). The infrared image is set to 320×320 resolution, normalized to the [0,1] interval, and converted to single-channel CHW format. Then, they are all uniformly transferred to the BEV space.

[0047] This embodiment aligns two-dimensional and three-dimensional images from multiple aspects, reducing the errors that may be caused by different image acquisition devices when acquiring images of the work area, and providing an accurate data foundation for subsequent image recognition.

[0048] In one embodiment, after the step of format aligning the two-dimensional and three-dimensional images of the acquired work area, the method further includes: Image enhancement processing is performed on both two-dimensional and three-dimensional images.

[0049] For example, voxel filtering can be applied to the LiDAR point cloud corresponding to the 3D image to reduce the point cloud density, thereby removing redundant and outlier point clouds and retaining only the effective point clouds.

[0050] Image enhancement is performed on RGB camera images. For example, in bright light scenes, adaptive histogram equalization (CLAHE) is used to suppress glare, while in low light scenes, the Retinex algorithm is used to enhance image brightness and contrast. In rainy scenes, "dark channel prior + guided filtering" is used to remove raindrops and reflections, ensuring that debris texture features are clear.

[0051] For infrared images, temperature compensation can be performed based on ambient temperature to eliminate the interference of ambient temperature on the special radiation of garbage.

[0052] In one embodiment, the two-dimensional data includes image data and infrared data, and the extraction result includes an image feature map, an infrared feature map, and a three-dimensional feature map. The step of performing feature extraction on the two-dimensional data and the three-dimensional data to obtain the extraction result includes: The image data is used to extract features using five convolutional layers and four pooling layers to obtain two-dimensional features. The two-dimensional features are then projected onto a fusion network using a viewpoint transformation matrix to obtain the image feature map in a preset format. Infrared features are obtained by using depthwise separable convolutions of a lightweight network to extract features from the infrared data, and the infrared features are projected onto the fusion network to obtain the infrared feature map in a preset format. The three-dimensional data is mapped onto the fusion network to obtain a sparse feature map. The sparse feature map is then processed by three sparse convolutional layers and two pooling layers to extract three-dimensional features. These three-dimensional features are then converted into a three-dimensional feature map in the preset format.

[0053] ResNet50 can be used as the backbone network to perform convolution and pooling operations on RGB camera images to extract the texture and color features of the images. Then, based on the camera intrinsics, the two-dimensional features are projected onto the BEV space through the view transformation matrix to generate a 128-channel visual BEV image feature map.

[0054] Infrared features, i.e. thermal radiation features, of infrared data are extracted using the depthwise separable convolution of the MobileNetV3-Small lightweight network. These features are then projected onto the BEV space to generate a 128-channel infrared BEV image feature map.

[0055] The three-dimensional data, i.e., the three-dimensional point cloud, is mapped to the BEV space to generate a sparse BEV feature map. Then, the Spconv (Sparse Convolution) network, i.e., three sparse convolutional layers and two pooling layers, is used to extract the spatial structure features of the point cloud and output a 128-channel three-dimensional BEV image feature map.

[0056] This embodiment extracts features from image feature maps, infrared feature maps, and 3D feature maps in different ways to obtain corresponding extraction results, and maps the extraction results to the BEV space, providing a data foundation for subsequent feature fusion.

[0057] In one implementation, the step of fusing the extracted results based on dynamic attention weights to obtain a fused feature map includes: A dynamic attention weight calculation model is constructed, and the lighting conditions for acquiring the two-dimensional image and the three-dimensional image are determined. The image feature map, the infrared feature map, the three-dimensional feature map, and the illumination conditions are input into the dynamic attention weight calculation model to obtain the first weight corresponding to the image feature map, the second weight corresponding to the infrared feature map, and the third weight corresponding to the three-dimensional feature map; The image feature map, the infrared feature map, and the three-dimensional feature map are fused according to the first weight, the second weight, and the third weight to obtain the fused feature map.

[0058] For channel alignment of 3D BEV features (L), image BEV features (V), and infrared BEV features (I), for example, the number of infrared feature channels can be increased from 64 to 128 by using 1×1 convolution.

[0059] The features L, V, I, and illumination conditions are then input into the dynamic attention weight calculation model, where the illumination conditions are used to adjust the weights of features L, V, and I. For example, in strong light scenes, the weight w_V of image feature V is increased; in occluded scenes, the weight w_L of 3D feature L is increased; and in material differentiation scenes, the weight w_I of infrared feature I is increased. Here, w_L + w_V + w_I = 1.

[0060] Then, according to the fusion formula Fusion_BEV = w_L×L + w_V×V + w_I×I, a 256-channel fused BEV feature map (size 150×150) is output. This fused feature map simultaneously contains the spatial structure, visual texture, and thermal radiation information of the waste.

[0061] This embodiment dynamically fuses image feature maps, infrared feature maps, and three-dimensional feature maps to obtain a fused feature map, which simultaneously possesses the spatial structure, visual texture, and thermal radiation information of waste, thereby improving the accuracy and efficiency of waste identification.

[0062] In one implementation, the step of performing target recognition on the fused feature map to obtain the target object in the fused feature map includes: Obtain the type of the target object, construct a corresponding detection head based on the type, and each type of target object corresponds to a detection branch in a detection head; The fused feature map is input into the detection head, and the recognition result of the fused feature map is output through the detection branch of the detection head.

[0063] After obtaining the fused feature map, a spatial attention mechanism can be used to highlight the target region features of the garbage and suppress the interference of the background. Then, the number of feature channels is reduced from 256 to 128 through 1×1 convolution + BatchNorm + ReLU, which reduces the computational load of subsequent detection modules while retaining the core features.

[0064] The dimensionality-reduced fused feature map is then input into the lightweight YOLOv9 detection head. The detection head in a neural network is a core component at the output end of models for tasks such as object detection and 3D perception. It is specifically responsible for transforming the general features extracted by the backbone network into the final prediction results required by the task.

[0065] For different types of trash, such as small, medium, and large trash, three detection branches are trained in the detection head, each corresponding to one of the three different types of trash. Each branch contains a convolutional + pooling + fully connected structure to output the BEV coordinates, detection boxes, and confidence scores of the trash. Then, the regression accuracy of the detection boxes is optimized through a loss function to reduce false detections of overlapping trash.

[0066] This embodiment improves the accuracy and efficiency of waste identification by training different detection branches for the detection head to detect and locate different types of waste.

[0067] In one implementation, the step of classifying the target object using a classifier based on the local features to obtain a classification result includes: For the detected target object, i.e. garbage, its local features in the fused feature map are extracted based on the detection box, and then input into the classifier to obtain the corresponding classification result.

[0068] The classification results include five categories: recyclables (7 categories including plastic bottles, aluminum cans, and glass jars), kitchen waste (4 categories including fruit peels and leftovers), hazardous waste (3 categories including batteries, light bulbs, and expired medicines), other waste (6 categories including cigarette butts and tissues), and bulky waste (2 categories including mattresses and sofas).

[0069] To improve the efficiency of waste disposal, the coordinates of waste in BEV space can be converted into three-dimensional coordinates in the coordinate system of the image acquisition device, and then converted into global latitude and longitude using GPS data from the image acquisition device.

[0070] In one embodiment, the method further includes optimizing the image recognition results using a loss function.

[0071] In the process of garbage identification, various losses occur, such as classification loss, regression loss, depth estimation loss, multimodal fusion loss, and multi-task loss. A joint loss function can be established based on these different losses, and then optimized to improve the accuracy of garbage identification.

[0072] Before format alignment of 2D and 3D images, object detection must be performed on both images. Format alignment can only be performed based on the object detection results. Therefore, supervised optimization can be performed using a classification loss function when detecting objects in 2D or 3D images.

[0073] When projecting 2D and 3D data onto the fusion network, supervision can be provided using a regression loss function. In this embodiment, the fusion network used for sampling is a BEV network. Projection alignment is achieved through target detection boxes. The regression loss function directly measures the positional difference between the predicted and ground truth boxes, offering simplicity, effectiveness, and relative insensitivity to outliers, thus helping to accurately regress the 3D position and size of the target.

[0074] In computer vision and image processing, most low-level and mid-level techniques directly involve pixel operations, such as image denoising, filtering, geometric transformation, and normalization. In these pixel-operation processes, supervision and optimization can be achieved by constructing depth estimation loss functions, such as binary cross-entropy loss.

[0075] When fusing the extracted results, supervision and optimization can be achieved by constructing a multimodal fusion loss function. For example, two detection heads can be introduced: one to obtain two-dimensional prediction results and one to obtain three-dimensional prediction results, as well as a fused prediction result. A fusion loss function can be constructed based on the losses corresponding to these three prediction results. By reasonably designing the weights of these losses, the effective fusion of multimodal information can be achieved.

[0076] In scenarios involving multiple tasks, such as simultaneously performing 3D object detection and occupancy grid prediction, a multi-task loss function can be used, such as Ltotal=α×Ldet+β×Locc+γ×Laff, where Ldet is the loss for the detection task, Locc is the loss for the occupancy grid prediction task, Laff is the loss for other auxiliary tasks, and α, β, and γ are the corresponding weight coefficients.

[0077] Please refer to Figure 4 In one embodiment, after step S203, the method further includes steps S205-S208.

[0078] S205. If the target object is a small target object and its local features cannot be extracted, then obtain the two-dimensional image corresponding to the small target object.

[0079] S206. The two-dimensional image is labeled to obtain the image coordinates corresponding to the small target object, and the image coordinates are mapped to a preset fusion space.

[0080] S207. Obtain the distribution density of the small target object in the preset fusion space, adjust the resolution of the fusion space according to the distribution density, and obtain the small target feature corresponding to the small target object in the preset fusion space.

[0081] S208. Enhance the features of the small target to obtain enhanced features, fuse the enhanced features with the three-dimensional features corresponding to the three-dimensional image to obtain a fusion result, and use the fusion result as the recognition result of the small target object.

[0082] Garbage identification algorithms based on multi-sensor fusion, such as BEVFusion, may suffer from the loss of information about small targets due to BEV feature compression, especially for small objects like small pieces of garbage. Therefore, when local features of small targets cannot be extracted, small target enhancement techniques can be used for small target identification.

[0083] First, after mapping the 2D image of the small target object to the BEV space, the resolution of the BEV space can be dynamically adjusted according to the distribution density of the small target object. For example, a high resolution (e.g., 0.1 m / grid) is used in dense areas, while a low resolution (e.g., 0.5 m / grid) is used in other areas, and a smooth transition between different resolution areas is achieved through boundary interpolation.

[0084] Secondly, this application can enhance the features of small targets. For example, it can use a residual attention mechanism to perform dual channel and spatial weighting on the high-frequency texture features of small targets. It can also construct a feature pyramid to fuse high-level semantic features and low-level detail features, thereby improving the discriminative power of small target features.

[0085] The small target features include texture features and semantic features. The step of enhancing the small target features to obtain enhanced features includes: High-frequency texture features are extracted from the texture features, and channel-weighted and spatial-weighted high-frequency texture features are applied to obtain optimized high-frequency texture features. Low-frequency texture features are extracted from the texture features, and low-pass filtering and multi-scale low-frequency fusion are performed on the low-frequency texture features to obtain optimized low-frequency texture features. The optimized high-frequency texture features and the optimized high-frequency texture features are then fused in the frequency domain to obtain enhanced texture features. High-level semantic features and low-level detail features are extracted from the semantic features, and the high-level semantic features and low-level detail features are fused together through a feature pyramid to obtain enhanced semantic features; The enhanced features are obtained by weighted fusion of the enhanced texture features and the enhanced semantic features.

[0086] After obtaining the enhanced features corresponding to the small target object, it is necessary to project the enhanced features into the BEV space and fuse them with the 3D features. At this time, the view frustum pooling algorithm can be used for optimization. For example, fine pooling can be performed only on the view frustum region containing the small target object to preserve pixel-level details, while simplified pooling such as average pooling can be used on the background region to reduce the amount of computation.

[0087] When processing 3D images of small target objects, the 3D point cloud of these objects may be sparse. Therefore, the PointPainting algorithm can be used to select high-confidence features from camera and infrared images and fill them into the 3D point cloud. Then, the RGB image features, infrared features, and 3D image features are fused using a dynamic weighting method to obtain the fused features of the small target object. The weights for dynamic weighting can be dynamically allocated based on point cloud density and depth error. Finally, object recognition and localization are performed on the fused features, and the small target objects are cleaned up based on the recognition and localization results.

[0088] In one implementation, the step of obtaining the two-dimensional image corresponding to the small target object includes: The small target detection model is used to extract targets from the two-dimensional image of the work area to obtain the two-dimensional image corresponding to the small target object; The training process of the small object detection model includes: A target detection model is constructed by inputting training images into the target detection model and constructing a weighted loss function to guide the training of the target detection model; the weighted loss function includes target classification loss, small target localization loss, and small target missed detection loss; The target detection model corresponding to the weighted loss function when the loss value is less than or equal to the preset loss value is taken as the small target detection model.

[0089] When detecting small objects in 2D images, the detection model can be trained using a weighted loss function. The loss function is Ltotal = λ1 × Lcls + λ2 × Lreg + λ3 × Lsize, where: Lcls is the classification loss (using Focal Loss, which improves the α parameter by 2-3 times for small objects / long-tailed categories); Lreg is the localization loss (using CIoU loss for small objects to enhance bounding box regression accuracy); and Lsize is the size penalty term (imposing a higher penalty coefficient for missed detections of small objects). This loss function is used for guided training to obtain the object detection model. Then, the object detection model is used to detect objects in the 2D image of the work area, thus extracting the 2D images corresponding to small objects from the work area.

[0090] After obtaining the 2D image corresponding to the small target object, dense samples of small targets can be generated through "cropping-scaling-pasting" to simulate different lighting scenarios (such as nighttime garbage recognition) and occlusion scenarios (such as paper scraps covered by leaves), thus expanding the diversity of the training set. Then, the small target detection model is trained using the training set, enabling the small target detection model to quickly identify small targets from different images.

[0091] This embodiment addresses the issue of small target objects failing to be properly identified by providing a method for enhancing and identifying them. This allows the invention to not only identify larger pieces of trash but also quickly identify and clean up small pieces of trash, thus improving the efficiency of trash disposal.

[0092] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of an image recognition device is given below. Please refer to [link / reference]. Figure 5 , Figure 5 This is a functional block diagram of an image recognition device provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the image recognition device provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The image recognition device 500 includes: The image alignment module 501 also needs to perform format alignment on the two-dimensional and three-dimensional images of the acquired work area to obtain the corresponding two-dimensional and three-dimensional data. The feature fusion module 502 is used to extract features from the two-dimensional data and the three-dimensional data to obtain extraction results, and to fuse the extraction results based on dynamic attention weights to obtain a fused feature map; The target recognition module 503 is used to perform target recognition on the fused feature map to obtain the target object in the fused feature map; The target classification module 504 is used to extract local features of the target object and use a classifier to classify the target object according to the local features to obtain a classification result.

[0093] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory shown is either stored in or embedded in the operating system (OS) of the image recognition device, and can be used by... Figure 1 The processor executes the commands. Meanwhile, the data and program code required to execute these modules can be stored in memory.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0095] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0096] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image recognition method, characterized in that, include: The format of the acquired 2D and 3D images of the work area is aligned to obtain the corresponding 2D and 3D data. Feature extraction is performed on the two-dimensional data and the three-dimensional data to obtain extraction results, and the extraction results are fused based on dynamic attention weights to obtain a fused feature map; Target recognition is performed on the fused feature map to obtain the target object in the fused feature map; Local features of the target object are extracted, and a classifier is used to classify the target object based on the local features to obtain a classification result.

2. The image recognition method according to claim 1, characterized in that, The two-dimensional data includes image data and infrared data, and the extraction result includes an image feature map, an infrared feature map, and a three-dimensional feature map. The step of extracting features from the two-dimensional data and the three-dimensional data to obtain the extraction result includes: The image data is used to extract features using five convolutional layers and four pooling layers to obtain two-dimensional features. The two-dimensional features are then projected onto a fusion network using a viewpoint transformation matrix to obtain the image feature map in a preset format. Infrared features are obtained by using depthwise separable convolutions of a lightweight network to extract features from the infrared data, and the infrared features are projected onto the fusion network to obtain the infrared feature map in a preset format. The three-dimensional data is mapped onto the fusion network to obtain a sparse feature map. The sparse feature map is then processed by three sparse convolutional layers and two pooling layers to extract three-dimensional features. These three-dimensional features are then converted into a three-dimensional feature map in the preset format.

3. The image recognition method according to claim 2, characterized in that, The step of fusing the extracted results based on dynamic attention weights to obtain a fused feature map includes: A dynamic attention weight calculation model is constructed, and the lighting conditions for acquiring the two-dimensional image and the three-dimensional image are determined. The image feature map, the infrared feature map, the three-dimensional feature map, and the illumination conditions are input into the dynamic attention weight calculation model to obtain the first weight corresponding to the image feature map, the second weight corresponding to the infrared feature map, and the third weight corresponding to the three-dimensional feature map; The image feature map, the infrared feature map, and the three-dimensional feature map are fused according to the first weight, the second weight, and the third weight to obtain the fused feature map.

4. The image recognition method according to claim 1, characterized in that, The step of performing target recognition on the fused feature map to obtain the target object in the fused feature map includes: Obtain the type of the target object, construct a corresponding detection head based on the type, and each type of target object corresponds to a detection branch in a detection head; The fused feature map is input into the detection head, and the recognition result of the fused feature map is output through the detection branch of the detection head.

5. The image recognition method according to claim 1, characterized in that, After the step of obtaining the target object in the fused feature map, the method further includes: If the target object is a small target object and its local features cannot be extracted, then the two-dimensional image corresponding to the small target object is obtained; The two-dimensional image is annotated to obtain the image coordinates corresponding to the small target object, and the image coordinates are mapped to a preset fusion space; The distribution density of the small target object in the preset fusion space is obtained, and the resolution of the fusion space is adjusted according to the distribution density to obtain the small target feature corresponding to the small target object in the preset fusion space; The small target features are enhanced to obtain enhanced features. The enhanced features are then fused with the corresponding three-dimensional features of the three-dimensional image to obtain a fusion result. The fusion result is used as the recognition result of the small target object.

6. The image recognition method according to claim 5, characterized in that, The step of fusing the enhanced features with the corresponding 3D features of the 3D image to obtain a fusion result includes: Obtain the confidence level corresponding to the small target feature, and fill the small target feature with a confidence level greater than the confidence threshold into the three-dimensional feature to obtain the fused three-dimensional feature; The enhanced features and the fused 3D features are weighted and fused to obtain the fusion result.

7. The image recognition method according to claim 5, characterized in that, The small target features include texture features and semantic features. The step of enhancing the small target features to obtain enhanced features includes: High-frequency texture features are extracted from the texture features, and channel-weighted and spatial-weighted high-frequency texture features are applied to obtain optimized high-frequency texture features. Low-frequency texture features are extracted from the texture features, and low-pass filtering and multi-scale low-frequency fusion are performed on the low-frequency texture features to obtain optimized low-frequency texture features. The optimized high-frequency texture features and the optimized high-frequency texture features are then fused in the frequency domain to obtain enhanced texture features. High-level semantic features and low-level detail features are extracted from the semantic features, and the high-level semantic features and low-level detail features are fused together through a feature pyramid to obtain enhanced semantic features; The enhanced features are obtained by weighted fusion of the enhanced texture features and the enhanced semantic features.

8. The image recognition method according to claim 5, characterized in that, The step of obtaining the two-dimensional image corresponding to the small target object includes: The small target detection model is used to extract targets from the two-dimensional image of the work area to obtain the two-dimensional image corresponding to the small target object; The training process of the small object detection model includes: A target detection model is constructed by inputting training images into the target detection model and constructing a weighted loss function to guide the training of the target detection model; the weighted loss function includes target classification loss, small target localization loss, and small target missed detection loss; The target detection model corresponding to the weighted loss function when the loss value is less than or equal to the preset loss value is taken as the small target detection model.

9. An image recognition device, characterized in that, include: The image alignment module also needs to perform format alignment on the two-dimensional and three-dimensional images of the acquired work area to obtain the corresponding two-dimensional and three-dimensional data. The feature fusion module is used to extract features from the two-dimensional data and the three-dimensional data to obtain extraction results, and to fuse the extraction results based on dynamic attention weights to obtain a fused feature map; The target recognition module is used to perform target recognition on the fused feature map to obtain the target object in the fused feature map; The target classification module is used to extract local features of the target object and use a classifier to classify the target object based on the local features to obtain a classification result.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the image recognition method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image recognition method as described in any one of claims 1-8.

Images