Adaptive target identification method based on multi-modal data fusion

Abstract

The invention discloses a self-adaptive target recognition method based on multi-modal data fusion, and relates to the technical field of target recognition, and the method comprises the following steps: S1, collecting a to-be-recognized image through a camera of a robot; s2, inputting feature vectors corresponding to pixel points of the to-be-recognized image into the target model, and determining dichotomy labels of the pixel points; s3, generating a feature matrix of the pixel points according to the global label matrix corresponding to the dichotomy labels of the pixel points; and S4, determining a target area according to the feature parameters corresponding to the feature matrix of the pixel points. According to the method, the model complexity and the classification precision are balanced through adaptive parameters, the problem of reasoning delay caused by a high-precision model is avoided, and the scene adaptability of the scheme is improved.

Description

Technical Field

[0001] This invention relates to the field of target recognition technology, and more specifically to an adaptive target recognition method based on multimodal data fusion. Background Technology

[0002] In the application scenarios of autonomous mobile robots (such as industrial AGVs, warehouse delivery robots, and factory inspection robots), obstacle target recognition during movement is one of the core technologies to ensure the safe, stable, and efficient movement of robots. The real-time performance, environmental adaptability, and accuracy of the recognition directly determine the robot's path planning efficiency, obstacle avoidance reaction speed, and operational safety in complex scenarios (such as avoiding collisions with factory equipment, material boxes, personnel, etc.).

[0003] Currently, existing technologies for obstacle target recognition during robot movement still suffer from the following common shortcomings: First, their environmental adaptability and anti-interference capabilities are limited. Most methods rely on single pixel features of visual images (such as color and texture) for recognition, but dynamic interference is common in robot movement scenarios. For example, the switching between strong and weak light in a factory area (backlighting at warehouse entrances and exits, dim environments during nighttime inspections) can cause visual feature distortion, or there may be reflections on obstacle surfaces (metal shelves, oil drums), or material obstruction (stacked cardboard boxes obscuring small obstacles). Relying solely on single visual features can easily lead to missed or false detections of obstacles. Secondly, there is a lack of coordination between local details and the global environment. Existing methods either focus on nearby local pixels while ignoring global planning at a distance (leading to insufficient early obstacle avoidance) or emphasize global environment modeling while losing nearby details (leading to missed detection of small and low obstacles). Thirdly, it is difficult to balance real-time performance and recognition accuracy. The operating rhythm of robot movement (such as the transportation speed of AGV sorting lines) requires a high recognition latency. However, in existing methods, high-precision recognition models are often computationally complex and have slow inference speeds, while lightweight models are prone to losing obstacle feature details, making it difficult to achieve an effective balance between high-speed movement and accurate recognition. Summary of the Invention

[0004] To address the above problems, this invention proposes an adaptive target recognition method based on multimodal data fusion.

[0005] The technical solution of this invention is: an adaptive target recognition method based on multimodal data fusion, comprising the following steps:

[0006] S1. Use the robot's camera to collect the image to be recognized;

[0007] S2. Input the feature vectors corresponding to the pixels of the image to be identified into the target model to determine the binary classification label of the pixels;

[0008] S3. Generate the feature matrix of the pixel based on the global label matrix corresponding to the binary classification label of the pixel;

[0009] S4. Determine the target region based on the feature parameters corresponding to the feature matrix of the pixel.

[0010] Furthermore, S2 includes the following sub-steps:

[0011] S21. Input the pixels of the image to be recognized into the label function to determine the initial label of each pixel;

[0012] S22. Train the target model using the initial labels and feature vectors of the pixels to determine the optimal parameters of the target model;

[0013] S23. Use the target with the determined optimal parameters to determine the binary classification label of the pixel.

[0014] The beneficial effects of the above-mentioned further scheme are as follows: In this invention, the initial label provides basic supervision information for the model, the model training further optimizes the classification logic, the parameters of the target model are optimized using the initial label and pixel feature vector as training data, the trained model is used to infer the pixel feature vector, and the final binary classification label is output.

[0015] Furthermore, in S21, the expression for the label function is:

[0016] ;

[0017] ;

[0018] in, Indicates the first The label value of each pixel. Indicates the first The pixel value of each pixel. Indicates the first The weight value of each pixel. Indicates exponentiation. This represents the total number of pixels in the image to be identified.

[0019] The beneficial effects of the above-mentioned further solutions are as follows: In this invention, pixel values ​​are converted into relative weights, exponential normalization amplifies the differences in pixel values, and the use of a "global weight mean" as a threshold can adapt to images with different brightness and contrast, such as images with bright and dark backgrounds, where the mean is automatically adjusted. The label function determines the classification label of a pixel (labeled as 1 and -1), which is the discrete label value for a binary classification task.

[0020] Furthermore, in S22, the expression for training the target model is:

[0021] ;

[0022] in, Indicates the first The label value of each pixel. These represent the weight parameters of the target model. This represents the bias parameters of the target model. Indicates exponentiation. This indicates adaptive parameter adjustment. This indicates taking the minimum value. Indicates the first The feature vector of each pixel.

[0023] The beneficial effects of the above-mentioned further solutions are: In this invention, This is a regularization term used to control model complexity and avoid overfitting. Even with a large number of pixel samples (images typically have tens of thousands of pixels), it prevents model fitting noise issues. It employs exponential loss instead of the traditional SVM hinge loss, resulting in a smoother penalty for classification errors and better adaptability to images with large pixel value fluctuations. The target model is trained with optimization involving all pixel samples. The optimization process involves summing the losses of all pixels, adjusting the sum to minimize the total loss and model complexity.

[0024] Furthermore, S3 includes the following sub-steps:

[0025] S31. Fill the pixel values ​​of the pixel and its eight neighboring pixels into a three-row, three-column matrix to form the global pixel matrix of the pixel.

[0026] S32. Fill the three-row, three-column matrix with the binary classification labels of the pixel and its eight neighboring pixels to form the global label matrix of the pixel;

[0027] S33. Perform convolution on the global pixel matrix and global label matrix of each pixel to obtain the feature matrix of each pixel.

[0028] The beneficial effects of the above-mentioned further solutions are as follows: In this invention, a single pixel is filled with the pixel values ​​of its eight surrounding pixels to form a 3×3 matrix, expanding the isolated information of a single pixel into the contextual information of a local region; similarly, a single pixel is filled with the binary classification labels of its eight neighbors to form a 3×3 matrix, expanding the local context of the labels; the convolution operation can capture the association between pixels and labels in the neighborhood (such as the collaborative features of pixel value changes and label changes at the target edge), and the generated feature matrix is ​​more discriminative.

[0029] Furthermore, in S33, the feature matrix of the pixel. The expression is:

[0030] ;

[0031] in, The global pixel matrix representing the pixels. A global label matrix representing pixels. This represents the convolution operation. This represents the activation function.

[0032] The beneficial effects of the above-mentioned further scheme are as follows: In this invention, the element-wise average of the global pixel matrix and the global label matrix is ​​first taken to balance the weights of the two matrices; the convolution result is mapped by the sigmoid function.

[0033] Furthermore, in S3, the rank of the feature matrix is ​​extracted. If the rank is greater than the average rank of all feature matrices, then the pixel is the target region.

[0034] The beneficial effects of this invention are as follows: By fusing pixel value features and classification label features to construct a multimodal feature matrix, and combining local neighborhood expansion to capture contextual information, this invention significantly enhances the resistance to complex interference in moving scenes. Whether it's visual feature distortion caused by strong backlighting in factory areas or dim nighttime environments, or situations such as reflections from metal obstacles or material occlusion, the multimodal fusion features can accurately distinguish obstacles from the background through the collaborative association of pixels and labels. In the feature extraction stage, this invention employs a lightweight convolution kernel and activation function design, reducing the complexity of feature calculation; simultaneously, by adaptively balancing model complexity and classification accuracy, it avoids the inference latency problem caused by high-precision models, improving the scenario adaptability of the solution. Attached Figure Description

[0035] Figure 1 This is a flowchart of an adaptive target recognition method based on multimodal data fusion. Detailed Implementation

[0036] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0037] like Figure 1 As shown, this invention provides an adaptive target recognition method based on multimodal data fusion, comprising the following steps:

[0038] S1. Use the robot's camera to collect the image to be recognized;

[0039] S2. Input the feature vectors corresponding to the pixels of the image to be identified into the target model to determine the binary classification label of the pixels;

[0040] S3. Generate the feature matrix of the pixel based on the global label matrix corresponding to the binary classification label of the pixel;

[0041] S4. Determine the target region based on the feature parameters corresponding to the feature matrix of the pixel.

[0042] In this embodiment of the invention, S2 includes the following sub-steps:

[0043] S21. Input the pixels of the image to be recognized into the label function to determine the initial label of each pixel;

[0044] S22. Train the target model using the initial labels and feature vectors of the pixels to determine the optimal parameters of the target model;

[0045] S23. Use the target with the determined optimal parameters to determine the binary classification label of the pixel.

[0046] The beneficial effects of the above-mentioned further scheme are as follows: In this invention, the initial label provides basic supervision information for the model, the model training further optimizes the classification logic, the parameters of the target model are optimized using the initial label and pixel feature vector as training data, the trained model is used to infer the pixel feature vector, and the final binary classification label is output.

[0047] In this embodiment of the invention, in S21, the expression of the tag function is:

[0048] ;

[0049] ;

[0050] in, Indicates the first The label value of each pixel. Indicates the first The pixel value of each pixel. Indicates the first The weight value of each pixel. Indicates exponentiation. This represents the total number of pixels in the image to be identified.

[0051] The beneficial effects of the above-mentioned further solutions are as follows: In this invention, pixel values ​​are converted into relative weights, exponential normalization amplifies the differences in pixel values, and the use of a "global weight mean" as a threshold can adapt to images with different brightness and contrast, such as images with bright and dark backgrounds, where the mean is automatically adjusted. The label function determines the classification label of a pixel (labeled as 1 and -1), which is the discrete label value for a binary classification task.

[0052] In this embodiment of the invention, in S22, the expression for training the target model is:

[0053] ;

[0054] in, Indicates the first The label value of each pixel. These represent the weight parameters of the target model. This represents the bias parameters of the target model. Indicates exponentiation. This indicates adaptive parameter adjustment. This indicates taking the minimum value. Indicates the first The feature vector of each pixel.

[0055] The beneficial effects of the above-mentioned further solutions are: In this invention, This is a regularization term used to control model complexity and avoid overfitting. Even with a large number of pixel samples (images typically have tens of thousands of pixels), it prevents model fitting noise issues. It employs exponential loss instead of the traditional SVM hinge loss, resulting in a smoother penalty for classification errors and better adaptability to images with large pixel value fluctuations. The target model is trained with optimization involving all pixel samples. The optimization process involves summing the losses of all pixels, adjusting the sum to minimize the total loss and model complexity.

[0056] In this embodiment of the invention, S3 includes the following sub-steps:

[0057] S31. Fill the pixel values ​​of the pixel and its eight neighboring pixels into a three-row, three-column matrix to form the global pixel matrix of the pixel.

[0058] S32. Fill the three-row, three-column matrix with the binary classification labels of the pixel and its eight neighboring pixels to form the global label matrix of the pixel;

[0059] S33. Perform convolution on the global pixel matrix and global label matrix of each pixel to obtain the feature matrix of each pixel.

[0060] The beneficial effects of the above-mentioned further solutions are as follows: In this invention, a single pixel is filled with the pixel values ​​of its eight surrounding pixels to form a 3×3 matrix, expanding the isolated information of a single pixel into the contextual information of a local region; similarly, a single pixel is filled with the binary classification labels of its eight neighbors to form a 3×3 matrix, expanding the local context of the labels; the convolution operation can capture the association between pixels and labels in the neighborhood (such as the collaborative features of pixel value changes and label changes at the target edge), and the generated feature matrix is ​​more discriminative.

[0061] In this embodiment of the invention, in S33, the feature matrix of the pixel points The expression is:

[0062] ;

[0063] in, The global pixel matrix representing the pixels. A global label matrix representing pixels. This represents the convolution operation. This represents the activation function.

[0064] The beneficial effects of the above-mentioned further scheme are as follows: In this invention, the element-wise average of the global pixel matrix and the global label matrix is ​​first taken to balance the weights of the two matrices; the convolution result is mapped by the sigmoid function.

[0065] In this embodiment of the invention, in step S3, the rank of the feature matrix is ​​extracted. If the rank is greater than the average rank of all feature matrices, then the pixel is the target region.

[0066] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. An adaptive target recognition method based on multimodal data fusion, characterized in that, Includes the following steps: S1. Use the robot's camera to collect the image to be recognized; S2. Input the feature vectors corresponding to the pixels of the image to be identified into the target model to determine the binary classification label of the pixels; S3. Generate the feature matrix of the pixel based on the global label matrix corresponding to the binary classification label of the pixel; S4. Determine the target region based on the feature parameters corresponding to the feature matrix of the pixel.

2. The adaptive target recognition method based on multimodal data fusion according to claim 1, characterized in that, S2 includes the following sub-steps: S21. Input the pixels of the image to be recognized into the label function to determine the initial label of each pixel; S22. Train the target model using the initial labels and feature vectors of the pixels to determine the optimal parameters of the target model; S23. Use the target with the determined optimal parameters to determine the binary classification label of the pixel.

3. The adaptive target recognition method based on multimodal data fusion according to claim 2, characterized in that, In step S21, the expression for the tag function is: ； ； in, Indicates the first The label value of each pixel. Indicates the first The pixel value of each pixel. Indicates the first The weight value of each pixel. This indicates exponentiation. This represents the total number of pixels in the image to be identified.

4. The adaptive target recognition method based on multimodal data fusion according to claim 2, characterized in that, In step S22, the expression for training the target model is: ； in, Indicates the first The label value of each pixel. These represent the weight parameters of the target model. This represents the bias parameters of the target model. This indicates exponentiation. This indicates adaptive parameter adjustment. This indicates taking the minimum value. Indicates the first The feature vector of each pixel.

5. The adaptive target recognition method based on multimodal data fusion according to claim 1, characterized in that, S3 includes the following sub-steps: S31. Fill the pixel values ​​of the pixel and its eight neighboring pixels into a three-row, three-column matrix to form the global pixel matrix of the pixel. S32. Fill the three-row, three-column matrix with the binary classification labels of the pixel and its eight neighboring pixels to form the global label matrix of the pixel; S33. Perform convolution on the global pixel matrix and global label matrix of each pixel to obtain the feature matrix of each pixel.

6. The adaptive target recognition method based on multimodal data fusion according to claim 5, characterized in that, In S33, the feature matrix of the pixel points The expression is: ； in, The global pixel matrix representing the pixels. A global label matrix representing pixels. This represents the convolution operation. This represents the activation function.

7. The adaptive target recognition method based on multimodal data fusion according to claim 1, characterized in that, In step S3, the rank of the feature matrix is ​​extracted. If the rank is greater than the average rank of all feature matrices, then the pixel is the target region.

Images