A multiscale companion robot target detection method, system and electronic device

Abstract

The application discloses a kind of multiscale accompanying robot target detection method, system and electronic equipment, method includes: constructing improved YOLOv5s network model, utilizes robot target detection dataset to train;The model trained is verified on test set, and the model after verification is used to carry out target detection to the image data collected;Improved YOLOv5s network adopts C2f module to replace C3 module in Backbone part, and SE attention mechanism module is added to the output end of C2f module, while introducing LSTM network, the feature map extracted is as the input of LSTM network;The mechanism of feature pyramid FPN and personal area network PAN is combined in the neck part of network, and the feature fusion capability of network is enhanced.The method improves the precision and efficiency of target detection of accompanying robot in the process of home service, greatly improves the work efficiency of accompanying robot.

Description

Technical Field

[0001] This invention relates to robotics technology, and in particular to a multi-scale companion robot target detection method, system, and electronic device. Background Technology

[0002] In recent years, the aging population has become a serious problem, urgently requiring the design of a more intelligent companion robot. Companion robots rely heavily on object detection algorithms. Currently, traditional object detection algorithms used in home environments are not robust enough. During detection, the complex home environment easily leads to false positives and false negatives, resulting in insufficient accuracy and real-time performance, failing to meet the speed and accuracy requirements of companion robots. Furthermore, there are currently insufficient datasets available for home detection. Deep learning-based object detection algorithms, due to their imperfect network structures, are prone to incomplete feature extraction and overfitting, further compromising detection accuracy and speed. Summary of the Invention

[0003] Purpose of the Invention: The purpose of this invention is to provide a multi-scale home target detection method for companion robots. This improves the accuracy of target detection in companion robots and enables real-time detection, thereby increasing the work efficiency of companion robots during their service process.

[0004] Technical solution: The present invention provides a multi-scale target detection method for companion robots, comprising the following steps:

[0005] An improved YOLOv5s network model was constructed and trained using a robot target detection dataset. The trained model was then validated on a test set, and the validated model was used to perform target detection on the collected image data.

[0006] The improved YOLOv5s network model is based on the YOLOv5s network model and includes the following improvements: In the feature extraction module Backbone, the C2fSE module replaces the C3 module, and an SE attention mechanism module is added to the output of the C2f module to form the C2fSE module, which uses the SE attention mechanism to perform weighted extraction of feature maps; At the same time, an LSTM network is also introduced into the feature extraction module Backbone. After the feature extraction layer extracts features, the extracted feature maps are used as input to the LSTM network. The LSTM network receives feature maps at each time step and learns the temporal relationship between features based on the previous feature maps and the feature maps of the current time step; The feature fusion module Neck adopts a combination of Feature Pyramid Network (FPN) and Personal Area Network (PAN) to enhance the network's feature fusion capability.

[0007] Furthermore, the improved YOLOv5s network model employs Gridmask data augmentation at its input.

[0008] Furthermore, a small detection head is added to the output of the improved YOLOv5s network model for detecting small targets.

[0009] Furthermore, the improved YOLOv5s network model adds a dice coefficient loss to the overall loss function. The formula for the overall loss function is:

[0010] Loss = βLoss box +γLsss cls +θLoss obj +ηLoss DC

[0011] Where Loss is the total loss function. box Loss is the bounding box loss function. cls For the category loss function, Loss obj For confidence loss, Loss DC denoted as the dice coefficient loss, and β, γ, θ, and η are the weighting coefficients of the bounding box loss, class loss, confidence loss, and dice coefficient loss, respectively.

[0012] Furthermore, the robot target detection dataset is the VOC dataset, which is generated as follows: First, image data is collected; second, image data is generated based on an improved WGAN adversarial generative network to expand the number of collected image data; then, the collected image data and the generated image data are packaged into a folder, and all images are labeled to generate corresponding XML files; finally, the XML files are converted into TXT files to generate the VOC dataset.

[0013] Furthermore, the improved WGAN adversarial generative network adds a corrector to the WGAN adversarial generative network and replaces the activation function in the network with the FRelu activation function to reduce the number of network parameters and enhance the realism of the generated image data.

[0014] Furthermore, the labelimg tool in Python is used to create a VOC dataset from the collected and generated image data.

[0015] Furthermore, before training the improved YOLOv5s network model using the robot object detection dataset, the robot object detection dataset needs to be scaled up, specifically as follows:

[0016] Gaussian pyramids are used to preprocess the robot target detection dataset, generating images of varying sizes from the image data. The saliency of the original image and the generated images of varying sizes is extracted, and feature learning is performed on the salient regions to obtain the salient regions.

[0017] Based on the same inventive concept, the present invention provides a multi-scale companion robot target detection system, comprising:

[0018] The model building and training unit is used to build an improved YOLOv5s network model and train it using a robot object detection dataset.

[0019] And a testing and object detection unit, used to validate the trained model on the test set, and use the validated model to perform object detection on the collected image data;

[0020] The improved YOLOv5s network model is based on the YOLOv5s network model and includes the following improvements: In the feature extraction module Backbone, the C2fSE module replaces the C3 module, and an SE attention mechanism module is added to the output of the C2f module to form the C2fSE module, which uses the SE attention mechanism to perform weighted extraction of feature maps; At the same time, an LSTM network is also introduced into the feature extraction module Backbone. After the feature extraction layer extracts features, the extracted feature maps are used as input to the LSTM network. The LSTM network receives feature maps at each time step and learns the temporal relationship between features based on the previous feature maps and the feature maps of the current time step; The feature fusion module Neck adopts a combination of Feature Pyramid Network (FPN) and Personal Area Network (PAN) to enhance the network's feature fusion capability.

[0021] Based on the same inventive concept, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the steps of the above-described multi-scale companion robot target detection method.

[0022] Beneficial effects: Compared with the prior art, the technical effects of the present invention are as follows: The present invention combines WGAN adversarial generative network with YOLOv5s network to improve the speed and accuracy of target detection in companion robots; (1) The method adds a corrector module to WGAN adversarial generative network and replaces the activation function of the network with FRelu function, which reduces the number of network parameters and improves the network's ability to generate images, constructs a dataset, enriches the types of datasets, and lays the foundation for subsequent YOLOv5s training; (2) The method uses feature learning of multi-scale salient regions of image pyramid to obtain the salient regions that need attention, which improves the detection accuracy; (3) The method changes the data augmentation method at the input end of YOLOv5s network to Gridmask data augmentation method, which enriches the types of datasets, improves the detection ability of small targets, and also enhances the robustness of the algorithm; (4) The method changes the C3 module of the Backbone part of YOLOv5s network to C2f module and adds SE attention mechanism to it, which improves the network (5) This method combines the LSTM network with the YOLOv5s network. After the YOLOv5s network extracts features through the feature extraction layer, the extracted feature map is used as the input of the LSTM network. Therefore, the LSTM can receive the feature map at each time step and learn the temporal relationship between features based on the previous features and the features of the current time step. By introducing historical sequence data into the LSTM, such as the feature map of the previous few frames, more contextual information can be provided, thereby improving the accuracy of target detection. (6) This method adds a smaller detection head at the output of YOLOv5s to enhance the network's ability to detect small targets. (7) This method improves the accuracy and efficiency of target detection in the home service process of the companion robot, greatly improving the working efficiency of the companion robot. (8) This method can encourage the model to optimize the segmentation results during the training process by putting the dice coefficient loss into the loss function of YOLOv5s, thereby improving the accuracy and precision of target segmentation. Attached Figure Description

[0023] Figure 1 This is a flowchart of a multi-scale companion robot target detection method in one embodiment of the present invention;

[0024] Figure 2 Flowchart for dataset creation and scaling;

[0025] Figure 3 A schematic diagram of the improved WGAN adversarial generative network structure;

[0026] Figure 4 This is a schematic diagram of the VOC dataset format structure;

[0027] Figure 5 A schematic diagram of the improved YOLOv5s network structure;

[0028] Figure 6 This is a schematic diagram of the C2fSE module structure;

[0029] Figure 7 This is a schematic diagram of an LSTM network structure. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings and specific examples.

[0031] This invention discloses a multi-scale target detection method for companion robots, comprising: constructing an improved YOLOv5s network model and training it using a robot target detection dataset; validating the trained model on a test set, and using the validated model to perform target detection on collected image data; wherein, the improved YOLOv5s network model is an improvement on the YOLOv5s network model, including: replacing the C3 module with a C2fSE module in the feature extraction module Backbone, adding an SE attention mechanism module to the output of the C2f module to form a C2fSE module, and using the SE attention mechanism to complete the weighted extraction of feature maps; simultaneously, an LSTM network is also introduced in the feature extraction module Backbone, after the feature extraction layer extracts features, the extracted feature map is used as the input of the LSTM network, the LSTM network receives the feature map at each time step, and learns the temporal relationship between features based on the previous feature map and the feature map of the current time step; the feature fusion module Neck adopts a combination of Feature Pyramid FPN and Personal Region Network PAN to enhance the feature fusion capability of the network.

[0032] like Figure 1 As shown, the target detection method for a multi-scale companion robot of the present invention comprises the following specific steps:

[0033] S1. Construct a robot target detection dataset, expand the constructed dataset, and then divide the expanded dataset into a training set and a test set;

[0034] Currently, there is no readily available robot object detection dataset for home applications. Therefore, a new dataset needs to be constructed. A large number of images were collected using a depth camera D435i. However, since more images are needed for later model training, the dataset needs to be further expanded. An improved WGAN adversarial generative network was used to generate images to expand the dataset and perform data augmentation.

[0035] The D435i depth camera can capture images up to 10 meters away and can transmit video at 90fps.

[0036] The WGAN Generative Adversarial Network consists of a generator module and a discriminator module. The generator continuously generates images by inputting signals, while the discriminator continuously judges whether the generated images are real. If they are not real, the results are returned to the generator module. The generator and discriminator continuously optimize each other. After several iterations, the generated images become more and more realistic. Finally, the generator can no longer determine that the image is fake and outputs the image.

[0037] The improved WGAN adversarial generative network (AGN) is an improvement upon the WGAN GAN, including the addition of a corrector module and the replacement of the activation function with FReLU. This reduces the number of parameters, enhances the realism of the images generated by the WGAN GAN, and addresses the gradient explosion problem that easily occurs during training, making the generated images increasingly closer to real images. A PyTorch environment was set up in PyCharm, parameters were configured, and the improved WGAN GAN was used to generate images, expanding the dataset and laying the foundation for subsequent YOLOv5s network training.

[0038] Improved WGAN adversarial generative network architecture, such as Figure 3 As shown, specifically: a noisy signal is input, and the discriminator determines whether it is true. If it is not true, a loss function is returned to the corrector. The corrector performs convolution and normalization operations to optimize the loss function and returns it to the generator. The generator starts generating images, and the discriminator then determines whether the images are true. The discriminator will distinguish between real and fake data. If the discrimination result is that the data is not all true, the corrector will penalize the generator module, causing the generator module to develop in the direction of generating real data, until it can fool the discriminator network.

[0039] The collected images and those generated by the improved WGAN adversarial generative network were packaged into a folder. Then, the labelimg tool in PyCharm was used to label the objects in all the images sequentially, completing the image annotation. The generated annotation files were saved in the same folder, forming XML files. A conversion script in Python was then used to convert the XML format to TXT format, generating the VOC dataset. The VOC dataset was then divided into training and testing sets. The VOC dataset format is as follows: Figure 4 As shown.

[0040] S2. Perform multi-scale salient region feature extraction. Use image pyramids to preprocess the image, generate images of different sizes, and learn the features of salient regions to obtain salient regions.

[0041] A Gaussian pyramid is a multi-scale representation of an image, or it can be described as generating images at different resolutions. The bottom layer of the image (layer 0 of the Gaussian pyramid), for example, a Gaussian kernel (5x5), is convolved. This convolution primarily processes even-numbered rows and columns, resulting in the next layer of the pyramid (layer 1). This convolution operation is repeated on this layer to obtain layer 2, and so on, until the Gaussian pyramid is obtained. After each operation, the M×N image is reduced to M / 2×N / 2, i.e., halved. The image of the Gaussian pyramid can be obtained using the following formula:

[0042]

[0043] Among them, G (n+1) (i,j) represents the new image, G n This represents the original image. W(m,n) = W(m) * W(n) is the sum of Gaussian convolutions of length 5, and the new image G is... (n+1) (i,j) is obtained by processing the original image G n It is obtained by sampling an even number of rows.

[0044] For the extraction of salient regions at multiple scales, an original image is denoted as A1. Gaussian pyramids are used for preprocessing to generate two images of different sizes. The salient values ​​of the two images are extracted and denoted as A2 and A3, respectively. Then, salient region feature learning is performed.

[0045] The flowcharts for steps S1 and S2 are as follows: Figure 2 As shown.

[0046] S3. Construct an improved YOLOv5s network model and train the model using the scaled training set.

[0047] like Figure 5 As shown, the improved YOLOv5s network model consists of four parts: input, backbone, neck, and output. The improved model reduces the number of network parameters while enhancing the network's feature extraction capabilities.

[0048] The input end uses Gridmask data augmentation, which involves randomly occluding or removing a discontinuous set of elements from an image. Essentially, it can be understood as a regularization method. This data augmentation method enriches the types of datasets, improves the detection capability for small targets, and enhances the robustness of the algorithm.

[0049] The backbone uses the C2f module instead of the C3 module, and adds an SE attention mechanism module to the output of the C2f module. Essentially, the C3 module in the backbone is replaced with a C2fSE module. The C2fSE module consists of the C2f module with an added SE attention mechanism module at the output. This reduces the number of network parameters while improving the network's feature extraction capabilities, making the network focus more on the target information that needs attention and ignoring irrelevant information, thus enhancing the network's attention to key information. The structure of the C2fSE module is as follows: Figure 6 As shown.

[0050] The backbone also incorporates an LSTM network, such as Figure 5 As shown, an LSTM network is added after the second and third C2fSE modules, combining the LSTM network with the YOLOv5s network. The improved YOLOv5s network extracts features through a feature extraction layer and uses the extracted feature maps as input to the LSTM network. Therefore, the LSTM can receive feature maps at each time step and learn the temporal relationships between features based on previous and current time step features. By introducing historical sequence data, such as feature maps from previous frames, into the LSTM, more contextual information can be provided, thereby improving the accuracy of object detection. The LSTM network is a traditional neural network structure used to process sequential data. While prone to gradient decay and gradient explosion during network training, it is well-suited for sequential data processing problems. The LSTM network structure is as follows: Figure 7 As shown.

[0051] The Neck section uses a combination of FPN and PAN, which greatly enhances the network's feature fusion capabilities.

[0052] A smaller target detection layer (i.e., a detection head) is added to the output end to detect targets that are easily overlooked due to their small size, thereby improving the network model's ability to detect smaller targets and thus improving the network's detection accuracy.

[0053] Finally, the dice coefficient loss is added to the total loss function of the network model. The formula for the total loss function is as follows:

[0054] Loss = βLoss box +rLoss cls +θLoss obj +ηLoss DC

[0055] Loss DC =1-DC

[0056]

[0057]

[0058]

[0059]

[0060] Where, Loss is the total loss function, Loss box Loss is the bounding box loss function. cls For the category loss function, Loss obj For confidence loss, Loss DC Let be the dice coefficient loss, β, γ, θ, and η be the weighting coefficients of the bounding box loss, class loss, confidence loss, and dice coefficient loss, respectively; DC be the dice coefficient; IoU be the intersection-over-union ratio of the ground truth bounding box and the predicted bounding box; A and B be the ground truth bounding box and the predicted bounding box, respectively; α be the parameter for balancing the ratio; v be the parameter describing the consistency of the aspect ratio between the predicted bounding box and the ground truth bounding box; and h be the parameter for balancing the bounding box and the predicted bounding box. gt w gt h and w are the height and width of the ground truth bounding box, respectively, and b and b' are the height and width of the predicted bounding box, respectively. gt ρ represents the center point of the predicted bounding box and the center point of the ground truth bounding box, respectively. 2 (b,b gt ) is the Euclidean distance between the center point of the predicted box and the center point of the ground truth box, and c is the diagonal length of the smallest rectangle containing the ground truth box and the predicted box.

[0061] Set up the training environment in PyCharm, configure the training parameters, put the training set into the improved YOLOv5s network model and start training. The pre-training weights are YOLOv5s.pt. After 400 iterations, the best model best.pt is finally obtained.

[0062] S4. Using the trained model best.pt as weights, validate it on the test set to check the detection effect of the model and finally achieve accurate detection; and use the validated model to perform target detection on the collected image data.

[0063] In summary, this invention combines the WGAN (Generative Adversarial Network) with the YOLOv5s network to improve the speed and accuracy of target detection in companion robots. This method adds a correction module to the WGAN generator and replaces the network's activation function with the FReLU function, reducing the number of network parameters while improving the network's ability to generate images. It also constructs and enriches the dataset, laying the foundation for subsequent YOLOv5s training. This method utilizes feature learning from multi-scale salient regions of the image pyramid to identify the salient regions of interest, thus improving detection accuracy. This method changes the data augmentation method at the YOLOv5s network input to Gridmask data augmentation, enriching the dataset, improving the detection capability for small targets, and enhancing the algorithm's robustness. This method replaces the C3 module in the backbone of the YOLOv5s network with a C2f module and adds an SE attention mechanism, increasing the network's focus on key information and thus improving the network's detection accuracy. This method combines an LSTM network with a YOLOv5s network. After the YOLOv5s network extracts features through a feature extraction layer, the extracted feature maps are used as input to the LSTM network. This allows the LSTM to receive feature maps at each time step and learn the temporal relationships between features based on previous and current time-step features. By incorporating historical sequence data, such as feature maps from previous frames, into the LSTM, more contextual information is provided, thereby improving the accuracy of object detection. A smaller detection head is added to the YOLOv5s output, enhancing the network's ability to detect small objects. By including the dice coefficients in the YOLOv5s loss function, this method encourages the model to optimize segmentation results during training, improving the accuracy and precision of object segmentation. This method improves the accuracy and efficiency of object detection in companion robots during home services, significantly enhancing the working efficiency of companion robots.

[0064] Based on the same inventive concept, the present invention provides a multi-scale companion robot target detection system, comprising:

[0065] The model building and training unit is used to build an improved YOLOv5s network model and train it using a robot object detection dataset.

[0066] And a testing and object detection unit, used to validate the trained model on the test set, and use the validated model to perform object detection on the collected image data;

[0067] The improved YOLOv5s network model is based on the YOLOv5s network model and includes the following improvements: In the feature extraction module Backbone, the C2fSE module replaces the C3 module, and an SE attention mechanism module is added to the output of the C2f module to form the C2fSE module, which uses the SE attention mechanism to perform weighted extraction of feature maps; At the same time, an LSTM network is also introduced into the feature extraction module Backbone. After the feature extraction layer extracts features, the extracted feature maps are used as input to the LSTM network. The LSTM network receives feature maps at each time step and learns the temporal relationship between features based on the previous feature maps and the feature maps of the current time step; The feature fusion module Neck adopts a combination of Feature Pyramid Network (FPN) and Personal Area Network (PAN) to enhance the network's feature fusion capability.

[0068] Based on the same inventive concept, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the steps of the above-described multi-scale companion robot target detection method.

Claims

1. A target detection method for a multi-scale companion robot, characterized in that, Includes the following steps: An improved YOLOv5s network model was constructed and trained using a robot target detection dataset. The trained model was then validated on a test set, and the validated model was used to perform target detection on the collected image data. The improved YOLOv5s network model is based on the YOLOv5s network model and includes the following improvements: In the feature extraction module Backbone, the C2fSE module replaces the C3 module, and an SE attention mechanism module is added to the output of the C2f module to form the C2fSE module, which uses the SE attention mechanism to perform weighted extraction of feature maps; At the same time, an LSTM network is also introduced into the feature extraction module Backbone, specifically: an LSTM network is added after the second and third C2fSE modules. After the feature extraction layer extracts features, the extracted feature maps are used as input to the LSTM network. The LSTM network receives feature maps at each time step and learns the temporal relationship between features based on the feature maps of the previous time step and the feature maps of the current time step; The feature fusion module Neck adopts a combination of Feature Pyramid Network (FPN) and Personal Area Network (PAN) to enhance the network's feature fusion capability.

2. The multi-scale companion robot target detection method according to claim 1, characterized in that, The improved YOLOv5s network model uses Gridmask data augmentation at its input.

3. The multi-scale companion robot target detection method according to claim 1, characterized in that, The improved YOLOv5s network model adds a small detection head to its output for detecting small targets.

4. The multi-scale companion robot target detection method according to claim 1, characterized in that, The improved YOLOv5s network model adds a dice coefficient loss to the overall loss function. The formula for the overall loss function is: ； in, For the total loss function, The bounding box loss function is... For the category loss function, For confidence loss, For the loss of dice coefficients, These are the weight coefficients for bounding box loss, class loss, confidence loss, and dice coefficient loss, respectively.

5. The multi-scale companion robot target detection method according to claim 1, characterized in that, The robot object detection dataset is the VOC dataset, which is generated as follows: First, image data is collected. Second, image data is generated based on an improved WGAN adversarial generative network to expand the number of collected image data. Then, the collected image data and the generated image data are packaged into a folder, and all images are labeled to generate corresponding XML files. Finally, the XML files are converted into TXT files to generate the VOC dataset.

6. The multi-scale companion robot target detection method according to claim 5, characterized in that, The improved WGAN adversarial generative network adds a corrector to the WGAN adversarial generative network. Specifically, when a noisy signal is input, the discriminator determines whether it is real. If it is not real, a loss function is returned to the corrector. The corrector performs convolution and normalization operations to optimize the loss function and returns it to the generator. The generator starts generating images, and the discriminator then determines whether the images are real. The discriminator distinguishes between real and fake data. If the discriminator determines that the data is not entirely real, the corrector will penalize the generator module, causing it to generate real data until it can fool the discriminator network. The activation function in the network is replaced with the FReLU activation function to reduce the number of network parameters and enhance the realism of the generated image data.

7. The multi-scale companion robot target detection method according to claim 5, characterized in that, The VOC dataset is created by combining the collected and generated image data using the labelimg tool in Python.

8. The multi-scale companion robot target detection method according to claim 1, characterized in that, Before training the improved YOLOv5s network model using the robot object detection dataset, the dataset needs to be scaled up, specifically as follows: Gaussian pyramids are used to preprocess the robot target detection dataset, generating images of varying sizes from the image data. The saliency of the original image and the generated images of varying sizes is extracted, and feature learning is performed on the salient regions to obtain the salient regions.

9. A multi-scale companion robot target detection system, characterized in that, include: The model building and training unit is used to build an improved YOLOv5s network model and train it using a robot object detection dataset. And a testing and object detection unit, used to validate the trained model on the test set, and use the validated model to perform object detection on the collected image data; The improved YOLOv5s network model is based on the YOLOv5s network model and includes the following improvements: In the feature extraction module Backbone, the C2fSE module replaces the C3 module, and an SE attention mechanism module is added to the output of the C2f module to form the C2fSE module, which uses the SE attention mechanism to perform weighted extraction of feature maps; At the same time, an LSTM network is also introduced into the feature extraction module Backbone, specifically: an LSTM network is added after the second and third C2fSE modules. After the feature extraction layer extracts features, the extracted feature maps are used as input to the LSTM network. The LSTM network receives feature maps at each time step and learns the temporal relationship between features based on the feature maps of the previous time step and the feature maps of the current time step; The feature fusion module Neck adopts a combination of Feature Pyramid Network (FPN) and Personal Area Network (PAN) to enhance the network's feature fusion capability.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the steps of the multi-scale companion robot target detection method according to any one of claims 1-8.

Images