Mobile robot and voice warning method
By integrating target human detection and behavior detection models into mobile robots, the challenge of identifying people's age and behavior in home environments has been solved, enabling effective voice warnings and enhancing security and detection capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QINGDAO HISENSE SMART LIFE TECH CO LTD
- Filing Date
- 2022-09-14
- Publication Date
- 2026-06-09
AI Technical Summary
Traditional image recognition technology cannot effectively identify a person's age and behavior in a home environment, resulting in an inability to provide appropriate voice warnings.
A mobile robot equipped with a camera and a voice player is used to identify the age and behavior of a target person through a trained target human detection model and behavior detection model, and to issue a voice warning when abnormal behavior is detected. The model includes a feature extraction network, a convolutional attention mechanism CBAMC3 module, and a behavior detection model. HSV data augmentation and mosaic data augmentation techniques are combined to improve the generalization ability and computational speed of the detection model.
It effectively identifies people's age and behavior in home environments, reduces the risk of abnormal behavior, enhances safety awareness, and improves the generalization ability and calculation speed of human body detection models.
Smart Images
Figure CN115439890B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing, and in particular to a mobile robot and a voice warning method. Background Technology
[0002] For mobile robots operating in home environments, they need to be able to identify the age, current behavior, and surrounding environment of individuals within their field of vision, and then provide corresponding services based on the identification results. However, because the situation of people and the surrounding environment are more complex in a home setting, and traditional image recognition technology is relatively simple in its recognition of information, it is not suitable for human identification in home environments. Therefore, there is an urgent need for a voice-based warning method applicable to home environments. Summary of the Invention
[0003] This application provides a mobile robot and a voice warning method, which can solve the problem in related technologies where voice warnings cannot be given to people in complex home environments. The technical solution is as follows:
[0004] On the one hand, a mobile robot is provided, the mobile robot including a controller, a camera, a mobile base and a voice player, the controller being used to: acquire a video stream of a target person captured by the camera, the target person being any person within the field of view of the camera;
[0005] The video stream is processed by a trained target human detection model to obtain human detection results, which indicate the age of the target person. The target human detection model includes a trained feature extraction network, which includes a downsampling module and a convolutional attention mechanism CBAMC3 module. The CBAMC3 module includes two convolutional layers, multiple mobile network MoblieNet layers, a feature fusion layer, and a convolutional attention layer.
[0006] If the target person is determined to be a specific person based on the human body detection results, the mobile base is controlled to move to follow the target person, and the video stream is processed by a trained behavior detection model to obtain the behavior of the target person. The specific person includes people whose age is below a first age threshold.
[0007] If the target person's behavior is abnormal, the voice player will be controlled to issue a voice warning.
[0008] Optionally, the target human detection model further includes a feature fusion network and a prediction network connected sequentially to the feature extraction network;
[0009] The controller is specifically used for:
[0010] Obtain a first sample image set, which includes multiple first sample images. Each first sample image in the multiple first sample images corresponds to a sample label, and the sample label indicates the age of the person in the corresponding first sample image.
[0011] Based on the first sample image set, the network parameters of the feature extraction network are trained to obtain trained network parameters;
[0012] A second sample image set is obtained, which includes multiple second sample images. Each second sample image in the multiple second sample images corresponds to a sample annotation information, and the sample annotation information indicates the imaging position of the person in the corresponding second sample image.
[0013] The network parameters of the feature extraction network are fixed to the trained network parameters. Based on the second sample image set, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human detection model.
[0014] Optionally, the controller is specifically used for:
[0015] Generate an image classification model, the image classification model including the feature extraction network, pooling layer and fully connected layer;
[0016] Based on the first sample image set, the image classification model is trained to obtain the trained network parameters.
[0017] Optionally, the controller is specifically used for:
[0018] The multiple second sample images are subjected to hue, saturation, and brightness HSV data enhancement and mosaic data enhancement to obtain multiple third sample images. The HSV data enhancement is used to improve the color diversity of the multiple second sample images, and the mosaic data enhancement is used to improve the diversity of shooting angles and shooting scenes of the multiple second sample images.
[0019] Based on the multiple second sample images and the multiple third sample images, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human body detection model, so that the target human body detection model can detect human bodies with diverse clothing colors, in different positions and scenes.
[0020] Optionally, the total number of the plurality of first sample images is greater than the total number of the plurality of second sample images, so as to improve the reliability of the network parameters of the feature extraction network.
[0021] On the other hand, a voice warning method is provided for use in a mobile robot, the method comprising:
[0022] Acquire a video stream of a target person, wherein the target person is any person within the shooting range;
[0023] The video stream is processed by a trained target human detection model to obtain human detection results, which indicate the age of the target person. The target human detection model includes a trained feature extraction network, which includes a downsampling module and a convolutional attention mechanism CBAMC3 module. The CBAMC3 module includes two convolutional layers, multiple mobile network MoblieNet layers, a feature fusion layer, and a convolutional attention layer.
[0024] If the target person is determined to be a specific person based on the human body detection results, then the target person is followed, and the video stream is processed by a trained behavior detection model to obtain the behavior of the target person. The specific person includes people whose age is below a first age threshold.
[0025] If the target person's behavior is abnormal, a voice warning will be issued.
[0026] Optionally, the method further includes:
[0027] Obtain a first sample image set, which includes multiple first sample images. Each first sample image in the multiple first sample images corresponds to a sample label, and the sample label indicates the age of the person in the corresponding first sample image.
[0028] Based on the first sample image set, the network parameters of the feature extraction network are trained to obtain trained network parameters;
[0029] A second sample image set is obtained, which includes multiple second sample images. Each second sample image in the multiple second sample images corresponds to a sample annotation information, and the sample annotation information indicates the imaging position of the person in the corresponding second sample image.
[0030] The network parameters of the feature extraction network are fixed to the trained network parameters. Based on the second sample image set, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human detection model.
[0031] Optionally, training the network parameters of the feature extraction network based on the first sample image set to obtain trained network parameters includes:
[0032] Generate an image classification model, the image classification model including the feature extraction network, pooling layer and fully connected layer;
[0033] Based on the first sample image set, the image classification model is trained to obtain the trained network parameters.
[0034] Optionally, the step of training the network parameters of the feature fusion network and the prediction network based on the second sample image set to obtain a trained target human detection model includes:
[0035] The multiple second sample images are subjected to hue, saturation, and brightness HSV data enhancement and mosaic data enhancement to obtain multiple third sample images. The HSV data enhancement is used to improve the color diversity of the multiple second sample images, and the mosaic data enhancement is used to improve the diversity of shooting angles and shooting scenes of the multiple second sample images.
[0036] Based on the multiple second sample images and the multiple third sample images, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human body detection model, so that the target human body detection model can detect human bodies with diverse clothing colors, in different positions and scenes.
[0037] Optionally, the total number of the plurality of first sample images is greater than the total number of the plurality of second sample images, so as to improve the reliability of the network parameters of the feature extraction network.
[0038] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the above-described voice warning method.
[0039] On the other hand, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the steps of the voice warning method described above.
[0040] The technical solution provided in this application can bring at least the following beneficial effects:
[0041] In the voice warning method provided in this application embodiment, the mobile robot can determine the age of the target person through a target human detection model, determine the current behavior of the target person through a trained behavior detection model, and issue a voice warning to the target person based on the current behavior. This reduces the risk factor of the target person engaging in abnormal behavior in a home environment and enhances the target person's safety awareness when engaging in abnormal behavior. Furthermore, the introduction of a convolutional attention layer into the convolutional attention mechanism module enhances the feature extraction capability of the target human detection model for global and semantic features of the image, thereby improving the generalization ability of the target human detection model. Adding multiple mobile network layers to the convolutional attention mechanism achieves lightweighting of the target human detection model, increases its computational speed, and improves human detection capabilities. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 This is a schematic diagram of the structure of a mobile robot provided in an embodiment of this application;
[0044] Figure 2 This is a flowchart of a voice warning method provided in an embodiment of this application;
[0045] Figure 3 This is a schematic diagram of the structure of YOLOv5 provided in an embodiment of this application;
[0046] Figure 4 This is a schematic diagram of the structure of a CBAMC3 module provided in an embodiment of this application;
[0047] Figure 5 This is a schematic diagram of the structure of a convolutional attention layer provided in an embodiment of this application;
[0048] Figure 6 This is a schematic diagram of a mobile robot application scenario provided in an embodiment of this application;
[0049] Figure 7 This is a schematic diagram of the structure of a voice warning device provided in an embodiment of this application. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0051] Before providing a detailed explanation of the voice warning method provided in the embodiments of this application, the application scenarios and implementation environment involved in the embodiments of this application will be introduced first.
[0052] First, the application scenarios involved in the embodiments of this application will be introduced.
[0053] In a home environment, mobile robots can detect the age and behavior of people within their field of vision in real time and provide personalized services based on this information. For example, a mobile robot can detect people in the home in real time. When it detects a child using kitchen knives and gas, it can issue a warning to remind the person to be careful, thus preventing potential dangers in daily life. Of course, other application scenarios are also involved in the embodiments of this application, which will not be elaborated here.
[0054] Next, the implementation environment provided in the embodiments of this application will be introduced.
[0055] The implementation environment of this application embodiment includes a mobile robot; please refer to [reference needed]. Figure 1 , Figure 1 This is a schematic diagram of the structure of a mobile robot provided in an embodiment of this application. As shown in the figure, the mobile robot includes: a controller 101, a camera 102, a mobile base 103, and a voice player 104; the controller 101 is communicatively connected to the camera 102, the mobile base 103, and the voice player 104. This communication connection can be wired or wireless, and this embodiment of the application does not limit it.
[0056] The controller 101 acquires a video stream of a target person captured by the camera 102. The target person refers to any person within the camera's field of view. The controller 101 processes the video stream using a trained target human detection model to obtain a human detection result, which indicates the age of the target person. If the human detection result determines that the target person belongs to a specific person, the controller 101 controls the mobile base 103 to move to follow the target person. The controller 101 also processes the video stream using a trained behavior detection model to obtain the behavior of the target person. Specific people include those whose age is below a first age threshold. If the target person's behavior is abnormal, the controller 101 controls the voice player 104 to issue a voice warning.
[0057] It should be noted that the controller can be a CPU (Central Processing Unit), the camera can be a regular camera, a night vision camera, or a 360-degree panoramic camera, the mobile base can be a motor-driven tire or track, and the voice player can be a smart speaker, a buzzer, etc. The embodiments of this application do not limit these.
[0058] Those skilled in the art should understand that the controller 101, camera 102, mobile base 103, and voice player 104 described above are merely examples. Other existing or future controllers, cameras, mobile bases, and voice players that are applicable to the embodiments of this application should also be included within the scope of protection of the embodiments of this application, and are hereby incorporated by reference.
[0059] It should be noted that the application scenarios and implementation environments described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of implementation environments, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0060] The voice warning method provided in the embodiments of this application will now be explained in detail.
[0061] Figure 2 This is a flowchart illustrating a voice warning method provided in an embodiment of this application, which is applied to a mobile robot. Please refer to... Figure 2 The method includes the following steps.
[0062] Step 201: Obtain the video stream of the target person, who is any person within the shooting range.
[0063] Mobile robots can film target individuals within their shooting range to obtain video streams of those individuals.
[0064] In a mobile robot comprising a controller, a camera, a mobile base, and a voice player, the controller controls the camera to capture a video stream of the target person, and then the controller acquires the video stream of the target person captured by the camera.
[0065] Step 202: Process the video stream using a trained target human detection model to obtain human detection results, which indicate the age of the target person. The target human detection model includes a trained feature extraction network, which includes a downsampling module and a Convolutional Block Attention Module (CBAM) C3 module. The CBAMC3 module includes two convolutional layers, multiple MobileNet layers, a feature fusion layer, and a convolutional attention layer.
[0066] The mobile robot inputs the video stream of the target person into a trained target human detection model to obtain the human detection results output by the target human detection model.
[0067] Optionally, the target human detection model may further include a feature fusion network and a prediction network connected sequentially to the feature extraction network. Thus, when the mobile robot processes the video stream, it first extracts features from the video stream using the feature extraction network to obtain human features. Then, the feature extraction network passes these human features to the feature fusion network, which fuses the human features and passes the fused human features to the prediction network. The prediction network then determines the human detection result of the video stream based on the fused human features.
[0068] The target human detection model is obtained by training the initial human detection model. Therefore, the network structure of the initial human detection model is the same as that of the target human detection model. Moreover, taking the initial human detection model as an example, the network structure of the initial human detection model can be a YOLOv5 structure, or other network structures; this embodiment does not limit this.
[0069] Taking the YOLOv5 architecture as an example, the initial human detection model includes a feature extraction network, a feature fusion network, and a prediction network connected in sequence. The feature extraction network includes a downsampling module and a CBAMC3 module. The CBAMC3 module includes two convolutional layers, multiple mobile network layers, a feature fusion layer, and a convolutional attention layer.
[0070] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a YOLOv5 structure provided in an embodiment of this application. As shown in the figure, the YOLOv5 structure includes a feature extraction network, a feature fusion network, and a prediction network connected in sequence. The feature extraction network is used to extract features from the image, the feature fusion network is used to fuse the features extracted by the feature extraction network, and the prediction network is used to predict the features fused by the feature fusion network to determine the human detection result.
[0071] The feature extraction network includes a Focus downsampling module, a K3 downsampling module, and a CBAMC3 module. The Focus downsampling module obtains a feature map by performing Focus downsampling on the input image. The K3 downsampling module performs K3 downsampling on this feature map. The CBAMC3 module is used for feature fusion, increasing the computational speed of the model, and extracting global location and semantic features from the K3 downsampling feature map.
[0072] The downsampling process described above refers to reducing the image size to decrease the number of sampling points, which means discarding some image information to improve the model's computation speed. It's important to note that Focus downsampling, while reducing the image size, increases the number of channels, thus preserving the image information completely. In other words, Focus downsampling does not lose image information; however, K3 downsampling does not completely preserve image information.
[0073] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a CBAMC3 module provided in an embodiment of this application. As shown in the figure, the CBAMC3 module includes two convolutional layers (Conv), multiple mobile network layers (mobilenet*n), a feature fusion layer (Concat), and a convolutional attention layer (CBAM). The convolutional layers are used for convolution processing; the multiple mobile network layers increase the computational speed of the model through depthwise separable convolutional structures in their network structure; the feature fusion layer is used for feature fusion; and the convolutional attention layer is used to extract global positional features and semantic features from the feature map.
[0074] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of a Convolutional Attention Layer (CBAM) provided in an embodiment of this application. As shown in the figure, the CBAM includes a Channel Attention Module (CAM), a first multiplier, a second multiplier, and a Spatial Attention Module (SAM). The CAM is used to extract semantic features from the input feature map; the first multiplier is used to multiply the unprocessed input feature map with the feature map processed by the CAM; the SAM is used to extract global positional features from the feature map; and the second multiplier is used to multiply the feature map output by the first multiplier with the feature map processed by the SAM.
[0075] Feature fusion networks include FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) structures.
[0076] The FPN structure includes the CBAMC3 module, a feature fusion module (Concat), an upsampling module, an SPPF (Spatial Pyramid Pooling-Fast) module, and a CBA module. The CBAMC3 module is used for feature fusion, increasing the model's computational speed, and extracting global location and semantic features from the feature map. The feature fusion module is used for feature fusion. The upsampling module upsamples the input feature map by enlarging it, increasing the number of sampling points, and thus increasing the extractable image features. The SPPF module converts feature maps of arbitrary size into feature vectors of fixed size. In the CBA module, C is a convolutional layer, B is a BN (Batch Normalization) layer, and A is an activation function. The convolutional layer performs convolutional processing, the BN layer performs batch normalization, and the activation function transforms simple linear mappings between data into complex nonlinear mappings.
[0077] The PAN structure includes a K3 downsampling module, a feature fusion module (Concat), and a CBAMC3 module. The K3 downsampling module performs K3 downsampling processing, the feature fusion module performs feature fusion, and the CBAMC3 module performs feature fusion, increases the computational speed of the model, and extracts global location and semantic features of the image.
[0078] The prediction network consists of three K1Conv modules. These three K1Conv modules perform convolutional processing on the features fused by the feature fusion network and determine the prediction result based on the convolutional processing result.
[0079] The initial human detection model can be trained according to the following steps (1)-(4) to obtain the trained target human detection model.
[0080] (1) Obtain a first sample image set, which includes multiple first sample images. Each first sample image in the multiple first sample images corresponds to a sample label, and the sample label indicates the age of the person in the corresponding first sample image.
[0081] The mobile robot first acquires multiple human images from the first image set and labels these multiple human images. The labeled multiple human images are called multiple first sample images, and each of the multiple first sample images corresponds to a sample label.
[0082] Based on the above description, the sample label indicates the age of the person in the corresponding first sample image. For example, the sample label can be one of the following four categories: child, youth, middle-aged, and elderly. Of course, these four categories are merely examples; in practical applications, other categories can be used depending on different needs, and this application does not limit this.
[0083] The first image set can be the ImageNet dataset or other datasets; this application embodiment does not limit this.
[0084] (2) Based on the first sample image set, the network parameters of the feature extraction network are trained to obtain the trained network parameters.
[0085] An image classification model is generated, which includes a feature extraction network, pooling layers, and fully connected layers. Based on the first sample image set, the image classification model is trained to obtain the trained network parameters.
[0086] Based on the above description, the initial human detection model includes a feature extraction network, a feature fusion network, and a prediction network. The feature extraction network is used to extract image features. Therefore, after the mobile robot acquires the first set of sample images, the feature fusion network and the prediction network in the initial human detection model are removed. Then, a pooling layer and a fully connected layer are sequentially connected after the feature extraction network to obtain the image classification model. In this way, based on the first set of image samples, the feature extraction network can be trained by training the image classification model, thereby obtaining the trained network parameters of the feature extraction network.
[0087] Since mobile robots need to provide personalized services based on a person's age, and feature extraction networks can extract features to represent a person's age, it is necessary to train the feature extraction network to enhance its feature extraction capabilities. Specifically, during the training of the image classification model based on the first sample image set, the feature extraction network learns the age of the people in the first sample images by extracting features from them, thereby training the network parameters.
[0088] (3) Obtain a second sample image set, which includes multiple second sample images. Each second sample image in the multiple second sample images corresponds to a sample annotation information, and the sample annotation information indicates the imaging position of the person in the corresponding second sample image.
[0089] The mobile robot obtains multiple human images from the second image set, and then uses an annotation tool to annotate these multiple human images. The annotated multiple human images are called multiple second sample images, and each of the multiple second sample images corresponds to a sample annotation information.
[0090] It should be noted that the second image set can be the CoCo dataset or other datasets, the annotation tool can be LabelImage, and the annotation information can be stored in an XML file. This application embodiment does not limit this.
[0091] (4) Fix the network parameters of the feature extraction network to the trained network parameters, and train the network parameters of the feature fusion network and the prediction network based on the second sample image set to obtain the trained target human detection model.
[0092] The network parameters of the feature extraction network are fixed to the trained network parameters. HSV data augmentation and mosaic data augmentation are performed on multiple second sample images to obtain multiple third sample images. Based on the multiple second sample images and multiple third sample images, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human detection model.
[0093] To ensure the reliability of the network parameters of the feature extraction network trained on the first sample image set, the total number of images in the first sample image set can be greater than the total number of images in the second sample image set. Thus, the network parameters trained on the first sample image set are more reliable than those trained on the second sample image set. Therefore, during the training of the feature fusion network and the prediction network, the network parameters in the feature extraction network are fixed to the trained parameters. This avoids changing the network parameters in the feature extraction network during the training of the feature fusion network and the prediction network, thereby preventing a decrease in the reliability of these network parameters.
[0094] Because clothing colors vary widely in home environments, and people may appear in different locations, accurate human detection requires HSV data augmentation and mosaic data augmentation on multiple second sample images to obtain multiple third sample images. Based on these second and third sample images, the network parameters of the feature fusion network and prediction network are trained. The trained target human detection model can then detect humans even in complex home environments. In other words, HSV data augmentation and mosaic data augmentation enhance the human detection capabilities of mobile robots in complex environments.
[0095] A mobile robot can perform at least one of the following processing methods on multiple second sample images: hue variation, saturation variation, and brightness variation, thereby achieving HSV data enhancement of the multiple second sample images to obtain images with different colors. In other words, HSV data enhancement refers to processing an image through at least one of the following methods: hue variation, saturation variation, and brightness variation, to obtain images with different colors.
[0096] For example, HSV data augmentation can be performed on multiple second sample images in the following seven ways: a. only changing the hue of the second sample image, b. only changing the saturation of the second sample image, c. only changing the brightness of the second sample image, d. changing both the hue and saturation of the second sample image, e. changing both the hue and brightness of the second sample image, f. changing both the saturation and brightness of the second sample image, and g. changing the hue, saturation, and brightness of the second sample image simultaneously.
[0097] The mobile robot can also perform at least one of the following processing steps—scaling, rotation, and translation—on multiple second sample images before stitching them together, thereby achieving mosaic data enhancement of the multiple second sample images to obtain images from different angles and in different scenes. That is, the aforementioned mosaic data enhancement refers to performing at least one of the following processing steps—scaling, rotation, and translation—on every four second sample images, and then stitching these four processed second sample images together using a common vertex to obtain images from different angles and in different scenes.
[0098] For example, when performing mosaic data augmentation on multiple second sample images, four of the second sample images can be processed in any of the following seven ways: a. scaling only the four second sample images, b. rotating only the four second sample images, c. translating only the four second sample images, d. scaling and rotating the four second sample images, e. scaling and translating the four second sample images, f. translating and rotating the four second sample images, g. scaling, rotating, and translating the four second sample images simultaneously.
[0099] In the process of training the initial human detection model to obtain the target human detection model, in order to obtain better training results, an optimizer can be used to train the initial human detection model for a certain number of epochs. For example, the number of epochs can be 300.
[0100] It should be noted that one epoch refers to the process of all data being fed into the network and completing one forward computation and backpropagation. The optimizer mentioned above can be the Adam optimizer or other optimizers. The 300 epochs mentioned above are only an example. In actual training, different numbers of epochs can be selected for training as needed. This application embodiment does not limit this.
[0101] Step 203: If the target person is determined to be a specific person based on the human detection results, then follow the target person and process the video stream using a trained behavior detection model to obtain the behavior of the target person. Specific people include those whose age is below the first age threshold.
[0102] If the mobile robot determines that the target person belongs to a specific group, it means that the mobile robot needs to detect the target person's behavior, so the mobile robot needs to follow the target person.
[0103] In a mobile robot comprising a controller, camera, mobile base, and voice player, if the controller determines that the target person belongs to a specific individual based on human detection results, the controller controls the mobile base to follow the target person and processes the video stream using a trained behavior detection model to obtain the target person's behavior.
[0104] In this process, the mobile robot can input a video stream into a trained behavior detection model to obtain the behavior of the target person output by the behavior detection model.
[0105] The aforementioned first age threshold is preset. For example, the first age threshold can be set to 6 years old, 8 years old, 10 years old, etc. In practical applications, it can be adjusted as needed. The aforementioned specific personnel can also include personnel older than the second age threshold. For example, the second age threshold can be 50 years old, 60 years old, 70 years old, etc. In practical applications, it can be adjusted as needed, and this application embodiment does not limit this. The behavior detection model is pre-trained, and this application embodiment does not limit the network structure of the behavior detection model.
[0106] Step 204: If the target person's behavior is abnormal, issue a voice warning.
[0107] If the target person's behavior is abnormal, it indicates that the abnormal behavior may cause potential danger. Therefore, the mobile robot needs to provide a voice warning to the target person to remind them to pay attention to safety.
[0108] In the case of a mobile robot that includes a controller, camera, mobile base and voice player, if the controller determines that the behavior of the target person is abnormal, the controller can issue a voice warning to the target person through the voice player.
[0109] The aforementioned abnormal behaviors can refer to various potentially dangerous behaviors of the target individual in a home environment. For example, please refer to... Figure 6 , Figure 6 This is a schematic diagram of an application scenario for a mobile robot provided in this application. As shown in the figure, the mobile robot first performs human body detection on the target person to determine the age of the target person. When the detected target person is a child, the mobile robot detects the current behavior of the target person and determines that the target person is currently using gas and knives. Then, the target robot will give a voice warning to the target person to remind the target person to pay attention to safety.
[0110] In summary, in the voice warning method provided in this application embodiment, the mobile robot can determine the age of the target person through the target human detection model, determine the current behavior of the target person through the trained behavior detection model, and issue a voice warning to the target person based on the current behavior. This reduces the risk factor of the target person engaging in abnormal behavior in a home environment and enhances the target person's safety awareness when engaging in abnormal behavior. Moreover, in the process of training the initial human detection model to obtain the target human detection model in this application embodiment, a convolutional attention layer is introduced into the convolutional attention mechanism module, which enhances the feature extraction capability of the target human detection model for global features and semantic features of the image, thereby improving the generalization capability of the target human detection model. Multiple mobile network layers are added to the convolutional attention mechanism, which realizes the lightweighting of the target human detection model, increases the computational speed of the target human detection model, and improves the human detection capability. In the process of training the feature extraction network based on the first sample image set, the feature extraction network learns the age of the people in the first sample image, which enhances the feature extraction network's feature extraction capability for people of different ages. Furthermore, in a home environment where people wear clothing of various colors, training the feature fusion network and prediction network with third-sample images augmented with HSV data enhances the feature fusion ability of the feature fusion network and the prediction ability of the prediction network in images of different colors, avoiding the problem of undetectable targets due to the variety of colors in the target image. Similarly, in a home environment where people are located in various corners, training the feature fusion network and prediction network with third-sample images augmented with mosaic data enhances the feature fusion ability of the feature fusion network and the prediction ability of the prediction network when people are in corners, avoiding the problem of undetectable targets when people are in corners.
[0111] Figure 7 This is a schematic diagram of the structure of a voice warning device provided in an embodiment of this application. This voice warning device can be implemented as part or all of a mobile robot by software, hardware, or a combination of both. Please refer to... Figure 7 The device includes: a video stream acquisition module 701, a video stream processing module 702, a behavior determination module 703, and a voice warning module 704.
[0112] The video stream acquisition module 701 is used to acquire the video stream of the target person being filmed, where the target person is any person within the filming range;
[0113] The video stream processing module 702 processes the video stream using a trained target human detection model to obtain human detection results, which indicate the age of the target person.
[0114] The target human detection model includes a trained feature extraction network, which includes a downsampling module and a convolutional attention mechanism CBAMC3 module. The CBAMC3 module includes two convolutional layers, multiple mobile network MoblieNet layers, a feature fusion layer, and a convolutional attention layer.
[0115] The behavior determination module 703 is used to follow the target person if the target person is determined to be a specific person based on the human detection results, and to process the video stream through a trained behavior detection model to obtain the behavior of the target person. The specific person includes people whose age is below an age threshold.
[0116] The voice warning module 704 is used to issue a voice warning if the behavior of the target person is abnormal.
[0117] Optionally, the voice warning device also includes:
[0118] The first acquisition module is used to acquire a first sample image set, which includes multiple first sample images. Each first sample image in the multiple first sample images corresponds to a sample label, and the sample label indicates the age of the person in the corresponding first sample image.
[0119] The first training module is used to train the network parameters of the feature extraction network based on the first sample image set, so as to obtain the trained network parameters.
[0120] The second acquisition module is used to acquire a second sample image set, which includes multiple second sample images. Each of the multiple second sample images corresponds to a sample annotation information, which indicates the imaging position of the person in the corresponding second sample image.
[0121] The second training module is used to fix the network parameters of the feature extraction network to the trained network parameters, and to train the network parameters of the feature fusion network and the prediction network based on the second sample image set to obtain the trained target human detection model.
[0122] Optionally, the first training module is specifically used for:
[0123] Generate an image classification model, which includes a feature extraction network, pooling layers, and fully connected layers;
[0124] Based on the first set of sample images, the image classification model is trained to obtain the trained network parameters.
[0125] Optionally, the second training module is specifically used for:
[0126] Multiple second sample images are subjected to HSV data enhancement for hue, saturation, and brightness, and mosaic data enhancement to obtain multiple third sample images. The HSV data enhancement is used to improve the color diversity of the multiple second sample images, and the mosaic data enhancement is used to improve the diversity of shooting angles and shooting scenes of the multiple second sample images.
[0127] Based on multiple second-sample images and multiple third-sample images, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human body detection model, so that the target human body detection model can detect human bodies with diverse clothing colors, different positions and scenes.
[0128] Optionally, the total number of multiple first sample images is greater than the total number of multiple second sample images to improve the reliability of the network parameters of the feature extraction network.
[0129] In summary, in the voice warning device provided in this application embodiment, the mobile robot can determine the age of the target person through the target human detection model, determine the current behavior of the target person through the trained behavior detection model, and issue a voice warning to the target person based on the current behavior. This reduces the risk factor of the target person engaging in abnormal behavior in a home environment and enhances the target person's safety awareness when engaging in abnormal behavior. In the process of training the initial human detection model to obtain the target human detection model in this application embodiment, a convolutional attention layer is introduced into the convolutional attention mechanism module, which enhances the feature extraction capability of the target human detection model for global features and semantic features of the image, thereby improving the generalization capability of the target human detection model. Multiple mobile network layers are added to the convolutional attention mechanism, which realizes the lightweighting of the target human detection model, increases the computational speed of the target human detection model, and improves the human detection capability. In the process of training the feature extraction network based on the first sample image set, the feature extraction network learns the age of the people in the first sample image, which enhances the feature extraction network's feature extraction capability for people of different ages. Furthermore, in a home environment where people wear clothing of various colors, training the feature fusion network and prediction network with third-sample images augmented with HSV data enhances the feature fusion ability of the feature fusion network and the prediction ability of the prediction network in images of different colors, avoiding the problem of undetectable targets due to the variety of colors in the target image. Similarly, in a home environment where people are located in various corners, training the feature fusion network and prediction network with third-sample images augmented with mosaic data enhances the feature fusion ability of the feature fusion network and the prediction ability of the prediction network when people are in corners, avoiding the problem of undetectable targets when people are in corners.
[0130] It should be noted that the voice warning device provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the voice warning device and the voice warning method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0131] In some embodiments, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the voice warning method described in the above embodiments. For example, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.
[0132] It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium, in other words, it can be a non-transient storage medium.
[0133] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.
[0134] That is, in some embodiments, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the voice warning method described above.
[0135] It should be understood that "at least one" as mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., are not necessarily different.
[0136] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first image set and the second image set involved in the embodiments of this application were obtained with full authorization.
[0137] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A mobile robot, characterized in that, The mobile robot includes a controller, a camera, a mobile base, and a voice player. The controller is used for: The system acquires a video stream of a target person captured by the camera, where the target person is any person within the camera's field of view. The video stream is then processed by a trained target human detection model to obtain a human detection result, which indicates the age of the target person. The target human detection model comprises a trained feature extraction network, a feature fusion network, and a prediction network connected in sequence. The feature extraction network includes a downsampling module and a convolutional attention mechanism (CBAMC3) module. The CBAMC3 module includes two convolutional layers, multiple mobile network (MoblieNet) layers, a feature fusion layer, and a convolutional attention layer. The convolutional attention layer includes a channel attention module, a first multiplier, a second multiplier, and a spatial attention module. If the target person is determined to be a specific person based on the human body detection results, the mobile base is controlled to move to follow the target person, and the video stream is processed by a trained behavior detection model to obtain the behavior of the target person. The specific person includes people whose age is below a first age threshold. If the behavior of the target person is abnormal, the voice player is controlled to issue a voice warning. The controller is further configured to: acquire a first sample image set, the first sample image set including a plurality of first sample images, each of the plurality of first sample images corresponding to a sample label, the sample label indicating the age of the person in the corresponding first sample image; An image classification model is generated, the image classification model including the feature extraction network, pooling layer and fully connected layer; the image classification model is trained based on the first sample image set to obtain trained network parameters; A second sample image set is obtained, comprising multiple second sample images. Each second sample image corresponds to a sample annotation, which indicates the imaging position of a person in the corresponding second sample image. The total number of the multiple first sample images is greater than the total number of the multiple second sample images to improve the reliability of the network parameters of the feature extraction network. The network parameters of the feature extraction network are fixed to the trained network parameters. The multiple second sample images are then subjected to HSV data enhancement (hue, saturation, brightness) and mosaic data enhancement to obtain multiple third sample images. The HSV data enhancement is used to improve the color diversity of the multiple second sample images, and the mosaic data enhancement is used to improve the diversity of the shooting angle and shooting scene of the multiple second sample images. Based on the multiple second sample images and the multiple third sample images, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human body detection model, so that the target human body detection model can detect human bodies with diverse clothing colors, in different positions and scenes.
2. A voice warning method, characterized in that, Applied to mobile robots, the method includes: A video stream of a target person is acquired, wherein the target person is any person within the shooting range; the video stream is processed by a trained target human detection model to obtain a human detection result, wherein the human detection result indicates the age of the target person; the target human detection model includes a trained feature extraction network, a feature fusion network, and a prediction network connected in sequence; the feature extraction network includes a downsampling module and a convolutional attention mechanism CBAMC3 module; the CBAMC3 module includes two convolutional layers, multiple mobile network MobileNet layers, a feature fusion layer, and a convolutional attention layer; the convolutional attention layer includes a channel attention module, a first multiplier, a second multiplier, and a spatial attention module. If the target person is determined to be a specific person based on the human body detection results, then the target person is followed, and the video stream is processed by a trained behavior detection model to obtain the behavior of the target person. The specific person includes people whose age is below a first age threshold. If the behavior of the target person is abnormal, then a voice warning is issued. Before processing the video stream using the trained target human detection model, the method further includes: acquiring a first sample image set, which includes multiple first sample images, each of which corresponds to a sample label indicating the age of the person in the corresponding first sample image; generating an image classification model, which includes the feature extraction network, pooling layers, and fully connected layers; training the image classification model based on the first sample image set to obtain trained network parameters; and acquiring a second sample image set, which includes multiple second sample images, each of which corresponds to a sample annotation, indicating the imaging position of the person in the corresponding second sample image. The total number of images is greater than the total number of the multiple second sample images to improve the reliability of the network parameters of the feature extraction network; the network parameters of the feature extraction network are fixed to the trained network parameters, and the multiple second sample images are subjected to hue, saturation, and brightness HSV data enhancement and mosaic data enhancement to obtain multiple third sample images. The HSV data enhancement is used to improve the color diversity of the multiple second sample images, and the mosaic data enhancement is used to improve the shooting angle and shooting scene diversity of the multiple second sample images; based on the multiple second sample images and the multiple third sample images, the network parameters of the feature fusion network and the prediction network are trained to obtain a trained target human detection model, so that the target human detection model can detect human bodies with diverse clothing colors and in different positions and scenes.