A target recognition method of a robot

Abstract

The application discloses a target recognition method of a robot, which comprises the following steps: collecting a panoramic image of the environment around the robot; performing enhancement preprocessing on the panoramic image for small targets to obtain a preprocessed image; performing semantic segmentation on the preprocessed image to extract a preset attention region, and dividing the attention region into multiple sub-images; performing multi-level cooperative recognition on each sub-image to determine whether the sub-image contains a target object and the category of the target object; the cooperative recognition comprises preliminary detection based on a convolutional neural network, and cross-modal semantic verification based on a multimodal large model and at least the preliminary detection result; and calculating the world coordinates of the target object in the working environment based on the position of the target object in the panoramic image and the positioning information of the robot. The application can solve the problems of low recognition accuracy of small-size targets and insufficient conversion accuracy of target pixel positions to world coordinates.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a target recognition method for robots. Background Technology

[0002] In the field of autonomous robot operations, such as sanitation cleaning and security patrols, accurately identifying specific targets in the environment (such as garbage and unusual objects) and obtaining their precise locations is crucial for achieving intelligent robot operations. Current technologies mostly employ visual recognition and positioning solutions to support this requirement, but significant shortcomings remain:

[0003] First, the features of small targets such as cigarette butts are easily obscured by the background, resulting in low target recognition accuracy;

[0004] Secondly, the accuracy of converting the pixel position of the target in the image into world coordinates usable by the robot is insufficient, making it difficult to support the robot's subsequent precise operations.

[0005] Therefore, there is an urgent need for a new target recognition method for robots to solve the problems of low recognition accuracy for small targets and insufficient accuracy in converting target pixel positions to world coordinates. Summary of the Invention

[0006] The purpose of this invention is to provide a target recognition method for robots, which addresses the aforementioned problems. This invention can solve the problems of low recognition accuracy for small targets and insufficient accuracy in converting target pixel positions to world coordinates.

[0007] The technical solution adopted in this invention is as follows:

[0008] A target recognition method for robots includes the following steps:

[0009] Step S1: Image acquisition. Acquire panoramic images of the robot's surrounding environment.

[0010] Step S2: Image enhancement. Perform enhancement preprocessing on the panoramic image targeting small targets to obtain a preprocessed image.

[0011] Step S3: Extraction of Region of Interest. The preprocessed image is semantically segmented to extract the preset region of interest, and the region of interest is divided into multiple sub-images.

[0012] Step S4: Collaborative recognition. Perform multi-level collaborative recognition on each sub-image to determine whether it contains a target object and to identify its category. The collaborative recognition includes preliminary detection based on a convolutional neural network, and cross-modal semantic verification by calling a multimodal large model based at least on the preliminary detection results.

[0013] Step S5: Target localization. Based on the position of the target object in the panoramic image and the robot's localization information, calculate the world coordinates of the target object in the working environment.

[0014] Furthermore, step S1 specifically includes:

[0015] S11. Multiple sets of raw images are simultaneously acquired through multiple vision sensors arranged around the robot;

[0016] S12. Perform optical distortion correction and time synchronization alignment on each group of original images;

[0017] S13. The corrected and aligned sets of original images are stitched together and fused to generate the panoramic image covering a 360-degree field of view around the robot.

[0018] Furthermore, the optical distortion correction includes radial distortion correction and tangential distortion correction; the time synchronization alignment specifically involves sending a hardware synchronization trigger signal to all visual sensors, causing all visual sensors to be exposed simultaneously within a set acquisition period, so as to acquire the original image with timestamp alignment.

[0019] Furthermore, step S2 includes the following sub-steps executed sequentially:

[0020] Step S21: Noise suppression. The panoramic image is filtered using an adaptive median filtering algorithm to suppress noise and preserve the edge features of small target objects.

[0021] Step S22: Detail enhancement. A multi-scale retinal enhancement algorithm is used to process the noise-suppressed image to improve the local details and contrast of small target objects.

[0022] Step S23: Image normalization. The image after detail enhancement is normalized to obtain the preprocessed image with consistent lighting conditions.

[0023] Furthermore, step S3 specifically includes:

[0024] S31. Semantic segmentation: The preprocessed image is processed using a semantic segmentation neural network model to output an initial binary mask image, in which pixels belonging to a preset region of interest are activated.

[0025] S32. Mask optimization: Perform morphological dilation and erosion operations on the initial binary mask image, and filter out connected regions with an area smaller than a threshold to obtain the optimized final mask image.

[0026] S33. Region cropping: Perform a dot multiplication operation between the preprocessed image and the final mask image to extract an image block containing only the region of interest.

[0027] S34. Divide the image block into multiple sub-images of the same size using a sliding window method.

[0028] Furthermore, step S4 specifically includes:

[0029] S41. First-level detection: The sub-image is input into a first convolutional neural network model, which outputs the location bounding boxes of one or more candidate target object regions and their preliminary categories and confidence scores, wherein the location bounding boxes, preliminary categories and confidence scores together constitute the preliminary detection results.

[0030] S42. Second-level classification: For each candidate target object region in the preliminary detection result, the corresponding image region is cropped from the sub-image according to its position bounding box, and the image region is input into the second convolutional neural network model for fine-grained classification to obtain a vector containing the probability values ​​of each category. The category corresponding to the highest probability value is taken as the fine classification result.

[0031] S43. Third-level verification: Based on the comparison between the confidence level of the fine classification result and the preset threshold, determine whether to trigger verification; if triggered, construct a text prompt that integrates the image region, the fine classification result, the preliminary category and confidence level in the preliminary detection result, the robot's current state and environmental context information, input it into the multimodal big model for analysis and judgment, and output the final recognition category of the target object by the big model.

[0032] Furthermore, the environmental context information includes at least one of the following: map information of the work area and task history information.

[0033] Furthermore, in step S5, the horizontal position coordinates of the target object in the three-dimensional world coordinate system are obtained by back-projection calculation using the two-dimensional pixel coordinates of the target object in the panoramic image, combined with the internal parameters of the vision sensor and the pose transformation matrix of the robot.

[0034] Furthermore, the target object is the garbage to be cleaned up.

[0035] Furthermore, the preset area of ​​interest includes the ground area and / or the area surrounding the waste container.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] This invention executes five core steps sequentially: image acquisition, image enhancement, region of interest extraction, collaborative recognition, and target localization. First, it acquires panoramic images of the robot's surrounding environment and performs enhancement preprocessing to optimize image quality for small targets. Then, it extracts preset regions of interest through semantic segmentation and divides the sub-image focusing recognition range. Next, it uses a multi-level collaborative recognition method of "preliminary detection by convolutional neural network + cross-modal semantic verification by multimodal large model" to determine whether the target object is included and to determine its category. Finally, it fuses the position of the target object in the panoramic image with the robot's positioning information to complete the calculation of the world coordinates of the target object in the working environment, forming a complete technical link of "image acquisition-optimization-recognition-localization".

[0038] This invention enables robots to accurately identify and spatially locate target objects in the work environment throughout the entire process. It enhances the recognition of small target features through image enhancement, reduces interference from irrelevant backgrounds by extracting the area of ​​interest, and balances detection efficiency and judgment accuracy through multi-level collaborative recognition. Target localization realizes the conversion of image pixel positions to actual world coordinates in the work environment, providing accurate target object category and spatial location information for the robot's subsequent operation decisions, and adapting to the operational needs of robots in various scenarios such as environmental sanitation cleaning and security inspection. Attached Figure Description

[0039] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0041] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0042] like Figure 1 As shown, this invention discloses a target recognition method for robots, comprising the following steps:

[0043] Step S1: Image acquisition. Acquire panoramic images of the robot's surrounding environment.

[0044] Step S2: Image enhancement. Perform enhancement preprocessing on the panoramic image targeting small targets to obtain a preprocessed image.

[0045] Step S3: Extraction of Region of Interest. The preprocessed image is semantically segmented to extract the preset region of interest, and the region of interest is divided into multiple sub-images.

[0046] Step S4: Collaborative recognition. Perform multi-level collaborative recognition on each sub-image to determine whether it contains a target object and to identify its category. The collaborative recognition includes preliminary detection based on a convolutional neural network, and cross-modal semantic verification by calling a multimodal large model based at least on the preliminary detection results.

[0047] Step S5: Target localization. Based on the position of the target object in the panoramic image and the robot's localization information, calculate the world coordinates of the target object in the working environment.

[0048] This invention employs five core steps sequentially: image acquisition, image enhancement, region of interest extraction, collaborative recognition, and target localization. First, it acquires panoramic images of the robot's surrounding environment and performs enhancement preprocessing to optimize image quality for small targets. Next, it extracts preset regions of interest through semantic segmentation and divides sub-images into focused recognition ranges. Then, it uses a multi-level collaborative recognition method—"preliminary detection via convolutional neural network + cross-modal semantic verification using a multi-modal large-model"—to determine whether a target object is present and to classify it. Finally, it fuses the target object's position in the panoramic image with the robot's localization information to calculate the target object's world coordinates within the working environment, forming a complete technical chain of "image acquisition-optimization-recognition-localization." This invention enables robots to accurately identify and spatially locate target objects in their working environment throughout the entire process. Image enhancement improves the recognizability of small target features, region of interest extraction reduces irrelevant background interference, multi-level collaborative recognition balances detection efficiency and judgment accuracy, and target localization converts image pixel positions to actual world coordinates in the working environment. This provides accurate target object category and spatial location information for subsequent robot operation decisions, adapting to the operational needs of robots in various scenarios such as sanitation cleaning and security inspection.

[0049] When robots are used in sanitation and cleaning, the target object is the garbage to be cleaned. The preset area of ​​interest includes the ground area and / or the area around the garbage container.

[0050] Furthermore, step S1 specifically includes:

[0051] Step S11: Simultaneously acquire multiple sets of raw images using multiple vision sensors arranged around the robot;

[0052] Step S12: Perform optical distortion correction and time synchronization alignment on each group of original images;

[0053] Step S13: Stitch and fuse the corrected and aligned sets of original images to generate the panoramic image covering a 360-degree field of view around the robot.

[0054] Image acquisition comprises three steps: simultaneous acquisition by multiple vision sensors, optical distortion correction and temporal synchronization alignment, and stitching and fusion. By employing a surround-type multi-sensor acquisition method, the limitations of a single sensor's field of view are overcome. First, distortion correction and temporal alignment are performed on the original images to eliminate geometric distortion and temporal misalignment. Then, stitching and fusion are used to generate a 360-degree panoramic image. Because of the surround-type multi-sensor acquisition, blind spots of a single vision sensor are eliminated. Optical distortion correction and temporal synchronization alignment ensure the geometric and temporal consistency of multiple sets of original images, avoiding ghosting and misalignment issues during stitching and fusion. The resulting 360-degree panoramic image possesses integrity and consistency, providing distortion-free and fully covered foundational image data for subsequent image enhancement, target recognition, and other steps.

[0055] Furthermore, the optical distortion correction in S12 includes radial distortion correction and tangential distortion correction;

[0056] The mathematical model for radial distortion correction is as follows:

[0057]

[0058]

[0059] Among them, the and The normalized image plane coordinates before correction are represented. and The coordinates represent the radial distortion corrected coordinates. express and The sum of squares (i.e.) ), the , , The radial distortion coefficients are obtained in advance using Zhang's calibration method;

[0060] The mathematical model for tangential distortion correction is as follows:

[0061]

[0062]

[0063] Among them, the and This represents the final coordinates after tangential distortion correction. , The tangential distortion coefficients are obtained in advance using Zhang's calibration method;

[0064] After correction, the final pixel coordinates are obtained through mapping using the intrinsic parameter matrix. The mapping formula is as follows: In the formula This is the camera intrinsic parameter matrix. , These are the pixel coordinates of the corrected image.

[0065] Optical distortion correction is refined into radial distortion correction and tangential distortion correction. Based on the distortion coefficients obtained by Zhang's calibration method, corresponding mathematical models are used to correct radial geometric distortion caused by lens optical characteristics and tangential geometric distortion caused by lens mounting deviations, respectively. Then, the corrected normalized coordinates are mapped to actual pixel coordinates through an intrinsic parameter matrix, achieving accurate geometric correction of the original image. Because a dedicated mathematical model is used to specifically correct radial and tangential distortions, and the coefficients obtained by Zhang's calibration method ensure the accuracy of the correction parameters, and the intrinsic parameter matrix completes the accurate mapping from normalized coordinates to pixel coordinates, the geometric distortion of the original image is effectively eliminated, pixel position deviations are corrected, and the geometric accuracy of the corrected image is guaranteed. This provides a unified pixel coordinate basis for the subsequent stitching and fusion of multiple images, avoiding stitching misalignment and contour distortion problems caused by image distortion.

[0066] Furthermore, the time synchronization alignment in S12 specifically involves sending a hardware synchronization trigger signal to all visual sensors, causing all visual sensors to be exposed simultaneously within a set acquisition period, ensuring that the image acquisition time difference is ≤1 millisecond, so as to acquire the original image with the timestamp aligned.

[0067] By sending a unified hardware synchronization trigger signal to all visual sensors, all sensors are forced to expose simultaneously within a set acquisition period. This strictly controls the image acquisition time difference between multiple sensors to within 1 millisecond, achieving precise timestamp alignment of multiple sets of original images. Because the hardware synchronization trigger signal ensures simultaneous exposure of all visual sensors and controls the acquisition time difference to within 1 millisecond, it eliminates time deviations caused by asynchronous acquisition, avoids image content misalignment caused by robot movement or target object displacement, and guarantees the temporal consistency of multiple sets of original images. This allows for precise matching of image content and pixel positions during subsequent stitching and fusion, improving the stitching alignment effect of panoramic images.

[0068] Furthermore, the splicing and fusion in S13 specifically includes:

[0069] Step S131, Feature Extraction and Matching: Extract feature points between images acquired by adjacent visual sensors and perform matching; use the FLANN matcher to calculate feature matching pairs, use the RANSAC algorithm to remove mismatched pairs, and retain valid matching pairs with an inlier ratio of ≥80%;

[0070] Step S132, Solving the transformation relationship: Based on the successfully matched feature point pairs, calculate the homography matrix between the adjacent images;

[0071] Step S133, Image Fusion: Based on the homography matrix, the adjacent images are transformed by perspective and aligned. The overlapping areas are then fused using a weighted fusion method to eliminate seams and generate a seamless panoramic image.

[0072] This invention breaks down the stitching and fusion process into three sub-steps: feature extraction and matching, transformation relationship solving, and image fusion. It uses the FLANN matcher to quickly achieve feature point matching, uses the RANSAC algorithm to eliminate mismatched pairs and retain a high proportion of valid matching pairs, solves the homography matrix based on the valid matching pairs to determine the perspective transformation relationship between images, and finally achieves image alignment through perspective transformation and performs weighted fusion on the overlapping areas.

[0073] Because it retains effective matching pairs with an inlier ratio of ≥80%, the homography matrix is ​​solved more accurately, thereby achieving precise perspective transformation alignment of adjacent images and avoiding image shift and ghosting after stitching. The weighted fusion method effectively eliminates the stitching seams in overlapping areas, ultimately generating a seamless 360-degree panoramic image with consistent visual effects, improving the integrity and visual consistency of the base image.

[0074] Furthermore, step S2 includes the following sub-steps executed sequentially:

[0075] Step S21: Noise suppression. The panoramic image is filtered using an adaptive median filtering algorithm to suppress noise and preserve the edge features of small target objects.

[0076] Step S22: Detail enhancement. A multi-scale retinal enhancement algorithm is used to process the noise-suppressed image to improve the local details and contrast of small target objects.

[0077] Step S23: Image normalization. The image after detail enhancement is normalized to obtain the preprocessed image with consistent lighting conditions.

[0078] This invention designs the image enhancement process into three progressive sub-steps executed sequentially: noise suppression, detail enhancement, and image normalization. First, noise suppression and small target edge preservation are achieved through adaptive median filtering. Then, the details and contrast of small targets are enhanced through a multi-scale retinal enhancement algorithm. Finally, image differences caused by illumination are eliminated through normalization processing, thus completing the layered optimization of the panoramic image.

[0079] By employing a progressive, hierarchical optimization strategy, adaptive median filtering preserves the edge features of small target objects while denoising. The multi-scale retinal enhancement algorithm specifically improves the detail recognition of small target objects, and image normalization eliminates brightness deviations under different lighting conditions. After three steps of processing, a preprocessed image with consistent lighting conditions and clear features is obtained, providing a high-quality image foundation for subsequent region of interest extraction and target recognition, and improving the detection sensitivity and robustness of subsequent algorithms.

[0080] Furthermore, the adaptive median filtering algorithm in step S21 is executed according to the following steps: defining the initial size and maximum size of the filtering window (the window size range is...). Initialize window size The algorithm calculates the median, minimum, and maximum values ​​of pixels within a window, and determines whether to increase the window size or output the median based on the relationship between the median and the minimum and maximum values, in order to preserve the edge details of small target objects while suppressing noise.

[0081] The adaptive median filtering algorithm defines the size range of the filtering window and dynamically adjusts it from the initial size. Based on the relationship between the median and maximum values ​​of pixels within the window, it decides whether to directly output the center pixel value or increase the window size, thereby achieving dynamic adaptation of the filtering window to the pixel features of different regions of the image.

[0082] Because the filtering window can be dynamically adjusted according to the image pixel features, rather than using a fixed window size, it effectively suppresses interference noise such as salt and pepper noise in panoramic images, while avoiding the blurring of small target object edge features caused by fixed window filtering. It fully preserves the edge details of small target objects, providing a clear target object outline basis for subsequent detail enhancement steps and ensuring the feature integrity of small target objects.

[0083] Furthermore, the multi-scale retinal enhancement algorithm in step S22 calculates the enhanced reflection component using the following formula. :

[0084]

[0085] Among them, the The pixel intensity of the input image is represented by the Represents the natural logarithm operation, the The total number of Gaussian kernel scales selected is represented by Σ, where Σ represents the summation operation. For scale indexing, This represents a two-dimensional convolution operation, the... Indicates the first A Gaussian kernel function, defined as follows: The The standard deviation of the Gaussian kernel is given by... This represents the natural exponential operation; after calculating the reflection component, it performs linear stretching to map the pixel value to the grayscale range of 0~255.

[0086] The multi-scale retinal enhancement algorithm estimates the luminance component by performing two-dimensional convolution operations on the noise-suppressed image using multi-scale Gaussian kernels, separates the reflection component and luminance component of the image using natural logarithm operation, averages the reflection component at multiple scales to obtain the enhanced reflection component, and finally maps the pixel values ​​to the standard grayscale range through linear stretching to enhance the detailed features of small target objects.

[0087] By using a multi-scale Gaussian kernel to achieve precise separation of the reflection component and the brightness component, multi-scale fusion improves the representation ability of local details of small target objects, and linear stretching further enhances the visual recognition of detail features, making small target objects with indistinct features clearer in the image, and significantly improving the detection sensitivity of small target objects in subsequent target recognition steps.

[0088] Furthermore, the standardization process in step S23 is Z-Score normalization, which is achieved through the following formula: in For the image with enhanced details, The mean of the image. The standard deviation of the image is . The preprocessed image is obtained after normalization.

[0089] The Z-Score normalization algorithm is employed. By calculating the pixel mean μ and standard deviation σ of the enhanced image, a normalization formula is used to convert each pixel value in the image into a standardized value. This eliminates the overall brightness deviation and pixel value fluctuations under different lighting conditions, achieving standardized grayscale processing. Because pixel value standardization is achieved through the mean and standard deviation, pixel value differences caused by uneven lighting and changes in ambient brightness in panoramic images are effectively eliminated. This ensures consistent lighting conditions in the preprocessed image, avoiding interference from lighting factors on subsequent semantic segmentation and object recognition algorithms, and improving the adaptability and robustness of subsequent algorithms under different lighting scenarios.

[0090] Furthermore, step S3 specifically includes:

[0091] Step S31: Semantic segmentation. The preprocessed image is processed using a semantic segmentation neural network model to output an initial binary mask image, in which pixels belonging to the preset region of interest are activated.

[0092] Step S32: Mask optimization. Morphological dilation and erosion operations are performed on the initial binary mask image, and connected regions with an area smaller than a threshold are filtered out to obtain the optimized final mask image.

[0093] Step S33: Region cropping. Perform a dot product operation between the preprocessed image and the final mask image to extract image blocks that contain only the region of interest.

[0094] Step S34: Divide the image block into multiple sub-images of the same size using the sliding window method.

[0095] The extraction of regions of interest (ROIs) is broken down into four sub-steps: semantic segmentation, mask optimization, region truncation, and partitioning. First, semantic segmentation generates an initial binary mask image to preliminarily extract ROIs. Then, morphological operations and connected component filtering optimize the mask quality. Next, dot product operations eliminate non-ROIs. Finally, a sliding window method is used to divide the ROI image block into standardized sub-images, achieving accurate extraction and standardization of ROIs. Because semantic segmentation and mask optimization accurately extract the predefined ROIs and eliminate irrelevant background areas, the computational load for subsequent target recognition is reduced. The sliding window method divides the ROI into sub-images of the same size, adapting to the input requirements of subsequent convolutional neural network models, avoiding recognition errors caused by inconsistent image sizes, and improving the efficiency and accuracy of collaborative recognition.

[0096] Furthermore, the semantic segmentation neural network model in step S31 is a convolutional neural network with a U-Net++ architecture. Its encoder contains multiple convolutional blocks for downsampling, and its decoder contains multiple transposed convolutional blocks for upsampling and concatenation with the corresponding layer features of the encoder. The output layer generates the initial binary mask image through a 1×1 convolutional layer and a sigmoid activation function. By using a U-Net++ convolutional neural network as the semantic segmentation model, its encoder extracts deep features from the preprocessed image through convolutional block downsampling. The decoder achieves multi-scale feature fusion by upsampling transposed convolutional blocks and concatenating them with the corresponding layer features of the encoder. The output layer generates a binary mask image through a 1×1 convolutional layer and a sigmoid activation function, accurately distinguishing pixels in the region of interest from those in the non-region of interest. The dense feature stitching advantage of the U-Net++ architecture enhances the extraction and fusion capabilities of multi-scale features. The 1×1 convolutional layer and the Sigmoid activation function achieve accurate binary partitioning of regions of interest and non-regions of interest. The generated initial binary mask image can accurately activate pixels in the preset regions of interest, providing a high-quality initial mask foundation for subsequent mask optimization steps and reducing the correction cost of mask optimization.

[0097] Furthermore, the morphological dilation operation in S32 uses a 3×3 rectangular structuring element to dilate the initial binary mask image, and the erosion operation uses a 3×3 rectangular structuring element to erode the dilated image. The area threshold is set to 100 pixels.

[0098] The expansion operation formula is as follows ,in The morphological dilation operator (SE) represents performing a dilation operation on the initial binary mask image M using the structuring element SE; the erosion operation formula is... ,in The morphological erosion operator represents applying the structuring element SE to the dilated mask image. Perform erosion operation; where M is the initial binary mask image, and SE is a 3×3 rectangular structuring element. Mask optimization uses a 3×3 rectangular structuring element and performs dilation operation. Fill in the tiny holes in the initial binary mask image, and then perform erosion operation. By eliminating burrs at the mask edges and filtering out connected regions smaller than 100 pixels, false small areas are removed, thus achieving morphological optimization of the mask image. The combination of "dilation-erosion" morphological operations specifically addresses the holes and burrs in the initial mask. The 100-pixel area threshold effectively eliminates false small regions, resulting in a final mask image with regular contours and accurate regions. This ensures that subsequent region extraction steps can extract complete and accurate regions of interest, avoiding omissions or mis-extractions of regions of interest due to mask defects.

[0099] Furthermore, the dot product operation in step S33 is to multiply the preprocessed image... With the final mask image Pixel-by-pixel multiplication yields a focused image containing only the region of interest. ,Right now .

[0100] Utilizing the characteristic of the final mask image that "pixel values ​​of 0 in non-interested areas and 1 in interested areas", the preprocessed image is... With the final mask image A pixel-by-pixel multiplication operation is performed to zero out the pixel values ​​of non-interested regions, retaining only the pixel values ​​of the interest regions. Because the multiplication operation utilizes the pixel characteristics of the mask image to accurately remove non-interested regions, only the focused image containing the preset interest region is extracted. This significantly reduces the interference of irrelevant background pixels on subsequent processing, focuses the distribution area of ​​the target object, and improves the processing efficiency of subsequent sub-image segmentation and target recognition.

[0101] Furthermore, in step S34, the sliding window method uses a window size of 64×64 pixels, with a step size of 32 pixels, to focus the image. The image is divided into multiple sub-images.

[0102] Using a fixed window size of 64×64 pixels, with a sliding step of 32 pixels, the image is focused. The image is divided into multiple sub-images of the same size by sliding sequentially upwards, thus standardizing the input units for target recognition. By dividing the focused image into sub-images of uniform size, it accurately matches the input size requirements of subsequent convolutional neural network models, avoiding recognition errors caused by inconsistent image sizes. This allows the collaborative recognition step to perform detection using standardized image units, improving the standardization and detection efficiency of collaborative recognition.

[0103] Furthermore, step S4 specifically includes:

[0104] S41. First-level detection: The sub-image is input into a first convolutional neural network model, which outputs the location bounding boxes of one or more candidate target object regions and their preliminary categories and confidence scores, wherein the location bounding boxes, preliminary categories and confidence scores together constitute the preliminary detection results.

[0105] S42. Second-level classification: For each candidate target object region in the preliminary detection result, the corresponding image region is cropped from the sub-image according to its position bounding box, and the image region is input into the second convolutional neural network model for fine-grained classification to obtain a vector containing the probability values ​​of each category. The category corresponding to the highest probability value is taken as the fine classification result.

[0106] S43. Third-level verification: Based on the comparison between the confidence level of the fine classification result and the preset threshold, determine whether to trigger verification; if triggered, construct a text prompt that integrates the image region, the fine classification result, the preliminary category and confidence level in the preliminary detection result, the robot's current state and environmental context information, input it into the multimodal big model for analysis and judgment, and output the final recognition category of the target object by the big model.

[0107] The collaborative recognition process is broken down into three progressive sub-steps: Level 1 detection, Level 2 classification, and Level 3 verification. Step S41 defines the location bounding box, preliminary category, and confidence level to form the initial detection result. Step S42 precisely crops the image region based on the location bounding box of the initial detection result and performs fine-grained classification. Step S43 determines whether to trigger verification based on the confidence level of the fine-grained classification result. If triggered, it integrates the image region, fine-grained classification result, core information from the initial detection result, robot state, and environmental context to construct a text prompt, which is then input into a multimodal large model to complete the final judgment. This achieves multi-level collaboration between "initial detection - fine-grained classification - cross-modal verification." Because the three steps are progressive and fully utilize the initial detection result, S41 provides clear processing objects and basic information for subsequent steps. S42 achieves fine-grained category determination of candidate target object regions. S43 supplements low-confidence results with multi-dimensional information and performs cross-modal verification through a multimodal large model, forming a logical closed loop for collaborative recognition. This effectively improves the accuracy of target object category determination and reduces misidentification and missed identification problems.

[0108] Furthermore, the environmental context information includes at least one of the following: map information of the work area and task history information. By incorporating at least one of the map information of the work area and task history information as environmental context information into the text prompts for the third-level verification, the pre-trained image-text multimodal processing language model is provided with background information about the robot's working environment. This allows the model to analyze and determine the target object category based on the actual situation of the work area. Because the environmental context information supplements the map and task history background of the work area, the cross-modal semantic verification of the multimodal model can be analyzed in conjunction with the robot's actual working scenario, avoiding one-sided judgments based solely on image information. This improves the rationality and accuracy of the third-level verification results and further ensures the reliability of the final target object identification category.

[0109] Furthermore, the first convolutional neural network model in step S41 is a YOLOv8 model, whose detection head adopts an Anchor-Free structure to directly predict the center coordinates, width and height, confidence level and class probability of the target object, and eliminate duplicate detection boxes through non-maximum suppression.

[0110] The formula for calculating the candidate region bounding box is as follows: , , , ,

[0111] In the formula , The coordinates of the target center are , Target width and height.

[0112] The YOLOv8 model is used as the first-level detection convolutional neural network model. Its anchor-free structure eliminates the need for pre-defined anchor boxes, directly predicting the center coordinates, width, height, confidence score, and class probability of the target object. The specific location of the candidate region bounding box is calculated using a dedicated formula, and then a non-maximum suppression algorithm is used to eliminate duplicate detection boxes, resulting in a unique candidate target object region. Because the anchor-free structure of the YOLOv8 model improves the detection capability and speed for small objects, the bounding box calculation formula achieves precise quantification of the candidate target object region location, and the non-maximum suppression algorithm effectively eliminates duplicate detection boxes, it can quickly and accurately obtain the location, preliminary class, and confidence score of the candidate target object region. This provides an accurate and unique candidate region foundation for subsequent second-level fine classification, improving the efficiency of fine classification.

[0113] Furthermore, the second convolutional neural network model in step S42 is a ResNet-50 model, which extracts deep features through multiple residual blocks containing residual connection structures, and outputs the vector containing the probability values ​​of each category through global average pooling and fully connected layers.

[0114] The formula for calculating the category probability distribution is: In the formula For the first The probability value of the category, The logits value is the output of the fully connected layer. The total number of target object categories is represented by the ResNet-50 model as the convolutional neural network model for the second-level classification. Deep features of candidate target object regions are extracted through residual blocks with residual connections, avoiding the gradient vanishing problem in deep networks. Global average pooling and fully connected layers then convert the deep features into category probability vectors. The probability distribution formula is used to accurately quantify the probability values ​​of each category, determining the fine-grained classification result. Because the residual connection structure solves the gradient vanishing problem in deep networks, it ensures the effective extraction of deep features. Global average pooling and fully connected layers achieve accurate conversion from features to category probabilities. The category probability distribution formula quantifies the confidence level of each category, improving the feature representation ability and accuracy of category determination for fine-grained classification, providing reliable fine-grained classification results for the third-level validation.

[0115] Furthermore, the forward propagation of the residual connection structure is achieved through the following formula: F out =ReLU(W2 ReLU(W1 F in +b1)+b2)+F in

[0116] Wherein, the F in Represents the input feature map, the Fout This represents the output feature map, where W1 and W2 represent the weight matrices of the convolutional layer. This represents a convolution operation, where b1 and b2 represent the bias vectors of the convolutional layer, ReLU(·) represents the rectified linear activation function, and +F at the end of the formula... in This represents a shortcut connection for identity mapping.

[0117] The residual connection structure extracts the input feature map F through two convolution operations and the ReLU activation function. in The residual features are then compared with the original input feature map F. in By performing identity mapping and adding the results, we obtain the output feature map F. out This approach enables the effective transfer of deep features, avoiding the vanishing gradient problem during the training of deep convolutional neural networks. The quick connection via identity mapping achieves the fusion of original input features and residual features, effectively solving the vanishing gradient problem in the ResNet-50 model during deep feature extraction. This ensures the model's ability to extract deep features from candidate target region images, resulting in more complete and distinctive deep features. It further optimizes the accuracy of fine-grained classification results, providing a reliable foundation for subsequent level-3 validation.

[0118] Furthermore, the text prompt constructed in step S43 includes a description of the image region, multiple top-ranked categories of the fine classification results, preliminary categories and confidence levels in the preliminary detection results, the robot's location information in the global map, and task scenario information; the multimodal large model is a pre-trained image-text multimodal processing language large model.

[0119] The constructed text prompts need to integrate image region descriptions, the top categories of fine classification results, the preliminary categories and confidence levels in the preliminary detection results, the robot's global map location information, and task scenario information. Furthermore, the multimodal large model used for verification is a pre-trained image-text multimodal processing language large model, providing comprehensive and multidimensional judgment criteria for cross-modal semantic verification.

[0120] Furthermore, in step S5, the horizontal position coordinates of the target object in the three-dimensional world coordinate system are obtained by back-projection calculation using the two-dimensional pixel coordinates of the target object in the panoramic image, combined with the internal parameters of the vision sensor and the pose transformation matrix of the robot.

[0121] In the target localization step, the two-dimensional pixel coordinates of the target object in the panoramic image are first determined. Then, the influence of imaging system parameters on the pixel coordinates is eliminated by combining the internal parameters of the vision sensor. The robot's pose transformation matrix is ​​used to achieve coordinate transformation from the image pixel coordinate system to the robot's carrier coordinate system, and then to the three-dimensional world coordinate system. The horizontal position coordinates of the target object are obtained through back projection calculation. Because the internal parameters of the vision sensor and the robot's pose transformation matrix are integrated, a multi-dimensional coordinate transformation from two-dimensional pixel coordinates to three-dimensional world coordinates is achieved. The back projection calculation completes the accurate mapping from the pixel position to the actual working environment position, determining the horizontal position of the target object in the actual working environment, and providing accurate spatial position information of the target object for the robot's subsequent operation decisions.

[0122] Furthermore, the specific formula for back projection calculation in step S5 is as follows:

[0123]

[0124] Among them, the Represents a homogeneous world coordinate column vector, the and The horizontal coordinates of the target object in the world coordinate system, the The column vector representing the homogeneous form of the panoramic image pixel coordinates, the and The pixel coordinates of the target object in the panoramic image, The matrix representing the inverse of the intrinsic parameter matrix of the vision sensor, the This represents the homogeneous transformation matrix from the robot's carrier coordinate system to the world coordinate system.

[0125] The panoramic image pixel coordinates and world coordinates of the target object are converted into homogeneous form. First, the homogeneous pixel coordinates are compared with the inverse matrix of the visual sensor intrinsic parameter matrix. Multiply to eliminate the influence of the vision sensor's intrinsic parameters on the imaging, and then combine with the homogeneous transformation matrix from the robot's coordinate system to the world coordinate system. Multiplication, through matrix multiplication operations, achieves spatial transformation of coordinates, ultimately extracting the horizontal coordinates from the homogeneous world coordinates. , Because the coordinate transformation is quantitatively calculated through homogeneous matrix operations, the inverse matrix of the intrinsic parameter matrix accurately eliminates parameter interference from the imaging system, and the homogeneous transformation matrix achieves precise coordinate system transformation, ensuring the mathematical accuracy of the coordinate transformation and significantly improving the calculation accuracy of the world horizontal coordinates of the target object in the working environment, making the robot's positioning of the target object more accurate.

[0126] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target recognition method for robots, characterized in that, Includes the following steps: Step S1: Image acquisition. Acquire panoramic images of the robot's surrounding environment. Step S2: Image enhancement. Perform enhancement preprocessing on the panoramic image targeting small targets to obtain a preprocessed image. Step S3: Extraction of Region of Interest. The preprocessed image is semantically segmented to extract the preset region of interest, and the region of interest is divided into multiple sub-images. Step S4: Collaborative recognition. Perform multi-level collaborative recognition on each sub-image to determine whether it contains a target object and to determine its category. The collaborative identification includes preliminary detection based on convolutional neural networks, and at least cross-modal semantic verification by calling a multimodal large model based on the preliminary detection results; Step S5: Target localization. Based on the position of the target object in the panoramic image and the robot's localization information, calculate the world coordinates of the target object in the working environment.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: S11. Multiple sets of raw images are simultaneously acquired through multiple vision sensors arranged around the robot; S12. Perform optical distortion correction and time synchronization alignment on each group of original images; S13. The corrected and aligned sets of original images are stitched together and fused to generate the panoramic image covering a 360-degree field of view around the robot.

3. The method according to claim 2, characterized in that, The optical distortion correction includes radial distortion correction and tangential distortion correction; The time synchronization alignment specifically involves sending a hardware synchronization trigger signal to all visual sensors, causing all visual sensors to be exposed simultaneously within a set acquisition period, in order to acquire the original image with timestamp alignment.

4. The method according to claim 1, characterized in that, Step S2 includes the following sub-steps executed sequentially: Step S21: Noise suppression. The panoramic image is filtered using an adaptive median filtering algorithm to suppress noise and preserve the edge features of small target objects. Step S22: Detail enhancement. A multi-scale retinal enhancement algorithm is used to process the noise-suppressed image to improve the local details and contrast of small target objects. Step S23: Image normalization. The image after detail enhancement is normalized to obtain the preprocessed image with consistent lighting conditions.

5. The method according to claim 1, characterized in that, Step S3 specifically includes: S31. Semantic segmentation: The preprocessed image is processed using a semantic segmentation neural network model to output an initial binary mask image, in which pixels belonging to a preset region of interest are activated. S32. Mask optimization: Perform morphological dilation and erosion operations on the initial binary mask image, and filter out connected regions with an area smaller than a threshold to obtain the optimized final mask image. S33. Region cropping: Perform a dot multiplication operation between the preprocessed image and the final mask image to extract an image block containing only the region of interest. S34. Divide the image block into multiple sub-images of the same size using a sliding window method.

6. The method according to claim 1, characterized in that, Step S4 specifically includes: S41. First-level detection: The sub-image is input into a first convolutional neural network model, which outputs the location bounding boxes of one or more candidate target object regions and their preliminary categories and confidence scores, wherein the location bounding boxes, preliminary categories and confidence scores together constitute the preliminary detection results. S42. Second-level classification: For each candidate target object region in the preliminary detection result, the corresponding image region is cropped from the sub-image according to its position bounding box, and the image region is input into the second convolutional neural network model for fine-grained classification to obtain a vector containing the probability values ​​of each category. The category corresponding to the highest probability value is taken as the fine classification result. S43. Third-level verification: Based on the comparison between the confidence level of the fine classification result and the preset threshold, determine whether to trigger verification; if triggered, construct a text prompt that integrates the image region, the fine classification result, the preliminary category and confidence level in the preliminary detection result, the robot's current state and environmental context information, input it into the multimodal big model for analysis and judgment, and output the final recognition category of the target object by the big model.

7. The method according to claim 6, characterized in that, The environmental context information includes at least one of the following: map information of the work area and task history information.

8. The method according to claim 1, characterized in that, In step S5, the horizontal position coordinates of the target object in the three-dimensional world coordinate system are obtained by back-projection calculation using the two-dimensional pixel coordinates of the target object in the panoramic image, combined with the internal parameters of the vision sensor and the pose transformation matrix of the robot.

9. The method according to claim 1, characterized in that, The target object is the garbage to be cleaned up.

10. The method according to claim 1, characterized in that, The preset areas of interest include the ground area and / or the area surrounding the waste container.

Images