Cow face recognition method and device based on RGB image and depth image

Abstract

The invention discloses a cattle face recognition method and equipment based on an RGB image and a depth image, and belongs to the technical field of biological image recognition. Comprising the following steps: acquiring and labeling data to obtain a labeled RGB data set and an unlabeled RGB data set; using a RetinaFace detection model to carry out cattle face position synchronous cutting on the RGB image and the corresponding depth image; and constructing an RGB and depth bimodal cattle face recognition model, sending the obtained tailored cattle face data set into two convolution feature extraction modules, respectively extracting RGB modal features and depth modal features, carrying out feature fusion through a feature pooling module, refining the features through an attention refining module, and obtaining a cattle recognition result. The method can reduce the influence of disturbing factors such as stains, and improves the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of biological image recognition, and more particularly to a method and device for cattle face recognition based on RGB images and depth images. Background Art

[0002] As beef, milk and other products account for a larger share of human needs, the cattle industry has grown and the number of cattle on cattle farms has increased. Insurance companies are increasingly demanding individual cattle identification for claims management and farm intelligent management. Cattle facial recognition, as a method of identifying cattle based on different facial features, plays a key role and has been widely used.

[0003] However, aspects such as stain occlusion and light environment will disrupt the image features of the cow's face, affect the feature extraction effect, and cause errors in the final recognition, which limits cow face recognition in practical applications. Among them, stain occlusion is also a problem that cannot be avoided and solved for pure RGB images.

[0004] Therefore, how to provide a cattle facial recognition method and device based on RGB images and depth images that can solve the above problems is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides a cattle facial recognition method and device based on RGB images and depth images, which solves the problems that pure RGB images cannot avoid stains, occlusion, light environment, etc., and proposes to increase the depth dimension so that the algorithm network can extract more features, and perform cattle face recognition with RGB and depth images to reduce the impact of disturbing factors such as stains.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention provides a method for cattle face recognition based on RGB images and depth images, comprising the following steps:

[0008] S1: Collect RGB images and corresponding depth images of cow faces at different angles, annotate the RGB images with cow face target frames, and obtain annotated RGB datasets and unannotated RGB datasets;

[0009] S2: Input the labeled RGB data set into the RetinaFace detection model training, use the trained RetinaFace detection model to perform cow face detection on the unlabeled RGB data set, and synchronously crop the cow face position of the depth image corresponding to each RGB image to obtain the cropped cow face data set;

[0010] S3: constructing an RGB and depth dual-modal cow face recognition model, wherein the RGB and depth dual-modal cow face recognition model includes: a convolutional feature extraction module, a feature pooling module, an attention refinement module, and a fully connected layer;

[0011] Among them, the obtained cropped cow face dataset is sent to two convolutional feature extraction modules to extract RGB modal features and deep modal features respectively. The RGB modal features and deep modal features are fused through the feature pooling module, and the features are refined through the attention refinement module to obtain the cow recognition results.

[0012] Preferably, the S1 comprises:

[0013] S11: Use a depth camera to shoot RGBD images of the cow's face from the front and from the left and right sides at 30 degrees, and obtain an RGB image and a corresponding depth image;

[0014] S12: Screening RGB images and depth images to form a preliminary data set;

[0015] S13: Perform data enhancement processing on the filtered RGB image and the corresponding depth image simultaneously;

[0016] S14: Annotate the cow face in the RGB image with a target box to obtain an annotated RGB dataset and an unannotated RGB dataset.

[0017] Preferably, the step of feeding the obtained cropped cow face dataset into two convolutional feature extraction modules to extract RGB modality features and depth modality features respectively, and fusing the RGB modality features and the depth modality features through a feature pooling module comprises:

[0018] S31: Send the RGB image and depth image in the cropped cow face dataset to two VGG-16 convolutional feature extraction modules to extract RGB modal features and depth modal features respectively, and obtain RGB convolution map and depth convolution map;

[0019] S32: The RGB modal features and the depth modal features are fused through a feature pooling module, and a pooling feature map is established through the feature pooling module.

[0020] Preferably, the step of refining the features through the attention refinement module to obtain the cattle identification result includes:

[0021] S33: The pooled feature map is used as the input of the attention refinement module, and the pooled features are refined through a fully connected layer with a tanh function activation;

[0022] S34: Convert the pooled features into an attention map using a convolutional layer with softmax activation;

[0023] S35: The attention weight is multiplied by the RGB modality feature embedding obtained in S31 and then passed through two fully connected layers, and then a softmax activation is performed to obtain the final recognition score;

[0024] S36: Find the corresponding cattle number according to the corresponding numerical value of the score to obtain the cattle identification result.

[0025] Preferably, the activation conversion formula for converting features into an attention map using a convolutional layer with softmax activation is as follows:

[0026] α att =softmax(Conv 1×1 (F sl ));

[0027] F sl =tanh((W sl ) T F dp );

[0028] Among them, Conv 1×1 is a convolutional layer with kernel size of 1×1 and 1 feature map, α att is the obtained attention map, F sl The activated feature tensor, F dp is the correlation feature information between the RGB image and the depth image, W sl Trainable weights for the fully connected layers in the attention refinement module.

[0029] Preferably, the attention weight is multiplied by the RGB modal feature embedding obtained in S31 and then passed through two fully connected layers, and then a softmax activation is performed to obtain the final recognition score, which is expressed as:

[0030]

[0031] Among them, W fc1 , W fc2 are the trainable weights of the two fully connected layers in the classifier, F RGB The RGB modality feature embedding output by the convolutional feature extraction module for the RGB channels in the convolutional feature extraction module.

[0032] Preferably, it also includes setting a loss function during the training process, and each loss function expression is:

[0033]

[0034]

[0035]

[0036] Among them, y RGB,c Represents the output vector of the input belonging to a certain category c among M categories, y RGB,c and Depth,c represents the output label of the corresponding input belonging to a category c among M categories, p RGB.c and p Depth.c is the probability score that is the output of the classifier.

[0037] Preferably, the overall loss function is:

[0038] L Total =L RGB +L D +L attention ;

[0039] Where, L Total is the overall loss function, L RGB is the RGB image loss function, L D is the depth image loss function, L attention is the attention loss function.

[0040] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a cattle facial recognition method based on RGB images and depth images when executing the computer program.

[0041] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a method and device for cow face recognition based on RGB images and depth images, which solves the problems that pure RGB images cannot avoid stains, light environment, etc., and proposes to increase the depth dimension so that the algorithm network can extract more features, perform cow face recognition with RGB and depth images, reduce the influence of disturbing factors such as stains, and improve recognition accuracy. The specific beneficial effects are as follows:

[0042] 1. The present invention uses the RetinaFace detection model to train the RGB image cow face pictures in the labeled data set, which is conducive to directly sending new cow face pictures into the network for cow face detection without labeling, which increases convenience and better meets practical application needs.

[0043] 2. The present invention increases the features of cow face images by integrating the depth dimension, so that the features of each cow can be better captured and distinguished, thereby improving the recognition effect of the recognition model; at the same time, through the assistance of information from the depth dimension, the difference problem of cow face stains can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0046] Figure 2 An RGB image of a cow's face taken by a depth camera provided in an embodiment of the present invention;

[0047] Figure 3 A depth map of a cow's face taken by a depth camera provided in an embodiment of the present invention;

[0048] Figure 4 A network architecture diagram of a RetinaFace detection model provided in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the structure of a feature extraction module provided in an embodiment of the present invention;

[0050] Figure 6 A diagram of the VGG-16 network architecture of the RGB and depth dual-modal cow face recognition model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The purpose of the present invention is to provide a method and device for cattle face recognition based on RGB images and depth images, the method comprising: S1: data acquisition: using an RGBD depth camera to collect images of cattle faces at different angles, annotating the RGB images with cattle face target frames, and obtaining annotated data sets. S2: cattle face detection uses a cattle face detection model to perform cattle face detection on the collected RGB images, and synchronously crop the RGB images and depth images. S3: Construct an RGB and depth dual-modal cattle face recognition model. First, the RGB and depth images are respectively sent to a convolutional neural network for feature extraction to obtain features of the two modalities, and then the RGB features and the depth features are effectively fused through a two-layer attention mechanism, and finally the features are further refined through a fully connected layer to complete individual cattle recognition. Provides a solution to the problems in the prior art that pure RGB images cannot avoid stain occlusion, light environment, etc.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] See also Figure 1 The embodiment of the present invention discloses a cattle face recognition method based on RGB image and depth image, comprising the following steps:

[0055] S1: Collect RGB images and corresponding depth images of cow faces at different angles, annotate the RGB images with cow face target frames, and obtain annotated RGB datasets and unannotated RGB datasets;

[0056] S2: Input the labeled RGB data set into the RetinaFace detection model training, use the trained RetinaFace detection model to perform cow face detection on the unlabeled RGB data set, and synchronously crop the cow face position of the depth image corresponding to each RGB image to obtain the cropped cow face data set;

[0057] S3: constructing an RGB and depth dual-modal cow face recognition model, wherein the RGB and depth dual-modal cow face recognition model includes: a convolutional feature extraction module, a feature pooling module, an attention refinement module, and a fully connected layer;

[0058] Among them, the obtained cropped cow face dataset is sent to two convolutional feature extraction modules to extract RGB modal features and deep modal features respectively. The RGB modal features and deep modal features are fused through the feature pooling module, and the features are refined through the attention refinement module to obtain the cow recognition results.

[0059] In a specific embodiment, S1 specifically includes using an RGBD depth camera to collect images of a cow's face at different angles, annotating the RGB image with a cow's face target frame, and obtaining an annotated data set. More specifically:

[0060] S11: Data collection: Use a depth camera to capture RGBD images of the cow's face from the front and from the left and right sides at 30 degrees, and obtain RGB images and depth images, such as Figure 2-3 shown.

[0061] S12: Classify the cows in the captured images, and select the cow face photos with clear shots and no other objects blocking the faces to initially form a data set.

[0062] S13: Data preprocessing: Taking the RGB image of each cow as the standard, manually annotate the cow face target box in the image.

[0063] S14: Image data enhancement: Perform data enhancement processing on the collected cow face RGB images and depth images at the same time, including image rotation, mirror flipping, affine and perspective transformation, to obtain the enhanced cow face dataset.

[0064] Specifically, in S12, the RGB image and the depth image are screened to form a preliminary data set, including using different methods to screen the RGB image and the depth image respectively. If the image is invalid, it will be screened out in pairs and not used. For RGB images, images with occlusion and unclear images are invalid images; for depth images, if the depth data is not clearly collected and the facial contour of the cow cannot be recognized by the naked eye through grayscale transformation, it is an invalid image.

[0065] In a specific embodiment, S2 cow face detection uses a cow face detection model to perform cow face detection on the collected RGB image, and synchronously crop the RGB image and the depth image. Specifically:

[0066] S21: Training of detection model: Use the enhanced dataset of manually annotated cow face target detection frames to train the RetinaFace detection model, which is conducive to directly sending new cow face pictures to the network for cow face detection without manual annotation. The specific backbone network diagram of the RetinaFace detection model is as follows: Figure 4 shown.

[0067] S22: Cow face detection: Use the trained RetinaFace model to detect cow faces, and crop the areas where cow faces are detected in the RGB and depth images to obtain the final cow face dataset.

[0068] In a specific embodiment, S3 constructs an RGB and depth bimodal cow face recognition model. Figure 6The VGG-16 network architecture diagram of the RGB and depth dual-modal cow face recognition model provided by the embodiment of the present invention is specifically as follows:

[0069] S31: The obtained cropped cow face dataset is fed into two convolutional feature extraction modules to extract RGB modality features and depth modality features respectively. The structure of the feature extraction module can be found in Figure 5 .

[0070] In a specific embodiment, the RGB image and the depth image are sent to two VGG-16 convolutional feature extractors to extract RGB modality features and depth modality features respectively to obtain RGB and depth convolution maps.

[0071] The loss functions are:

[0072]

[0073]

[0074] where y RGB,c and Depth,c represents the output label of the corresponding input belonging to a certain category c among M possible categories, p RGB.c and p Depth.c is the probability score after passing the classifier, L RGB When it is a convolutional feature extraction module, after feature extraction, the RGB single channel sends the RGB feature map to the classifier alone and generates a loss function with the true label; L D When it is a convolutional feature extraction module, the loss function generated by the deep single channel when the deep feature map is sent to the classifier alone after feature extraction and the true label;

[0075] S32: Feature pooling module. Combined with the extracted feature map, a pooled feature map is established through the feature pooling module. Here, the RGB model features and the depth modality features are fused through the feature pooling module. After fusion, both RGB features and depth features are present, thus forming a feature map.

[0076] S321: These pooled features are fed into the attention refinement module to generate attention to the features extracted from the RGB convolution branch.

[0077] S322: Multiply the attention weights derived from the depth-guided attention mechanism with the RGB feature map extracted from the convolutional neural network to obtain the final set of salient features.

[0078] S323: Use the dot product as the similarity measure between the two feature vectors to create a pooled feature space. The pooled feature space can be calculated as F dp =tanh((W1) T FRGB )⊙tanh((W2) T F D ).

[0079] Among them, F RGB and F D They are the RGB and depth convolution feature maps obtained by the convolution module, ⊙ is the dot multiplication operation, W1 and W2 are the trainable weights of RGB and depth, and F dp It is the correlation feature information between RGB and depth mode.

[0080] S33: The pooled feature map is used as the input of the attention refinement module, and the pooled features are refined through a fully connected layer with tanh function activation.

[0081] S331: Tensor F generated by the feature pooling module dp As the input of the attention refinement module. This module first refines the pooled features through a fully connected layer with a tanh function activation, inputting the tensor to add nonlinearity to the representation. The activated feature tensor is named F sl : F sl =tanh((W sl ) T F dp ), where W sl is the trainable weight of the fully connected layer in the attention refinement module, which is used to calculate the M*M*1 dimension attention map ( Figure 6 M*M*1 purple figure in the attention module).

[0082] S34: Use a convolutional layer with softmax activation to transform F sl Converted to an attention map, the activation conversion formula is as follows: α att =softmax(Conv 1×1 (F sl )).

[0083] Among them, Conv 1×1 is a convolutional layer with kernel size of 1×1 and 1 feature map, and α att is the resulting refined attention map.

[0084] The loss function is where y RGB,c represents the output vector of the input belonging to a certain category c among M possible categories, p RGB.c and p Depth.c is the probability score as the output of the classifier, L attention The loss function generated by the RGB modality and the depth modality after feature pooling fusion and attention refinement module are sent to the classifier and the true label.

[0085] S35: The attention weight is multiplied by the RGB embedding and passed through two fully connected layers, and then a softmax activation is performed to obtain the final recognition score.

[0086] The final recognition score function is:

[0087]

[0088] Where W fc1 , W fc2 are the trainable weights of the two fully connected layers in the classifier.

[0089] The overall loss function is: L Total =L RGB +L D +L attention .

[0090] S36: According to the output result of the scoring function, the corresponding cattle number can be found according to the corresponding numerical value, and finally the cattle identification is realized.

[0091] Specifically, this embodiment acquires data by: using an RGBD depth camera to acquire images of cow faces at different angles, annotating the RGB images with cow face target frames, and obtaining annotated data sets. Cow face detection uses a cow facial detection model to perform cow face detection on the acquired RGB images, and synchronously crop the RGB images and depth images. An RGB and depth dual-modal cow face recognition model is constructed. First, the RGB and depth images are respectively sent to a convolutional neural network for feature extraction to obtain features of the two modalities, and then the RGB features and the depth features are effectively fused through a two-layer attention mechanism. Finally, the features are further refined through a fully connected layer to complete individual cow recognition. The present invention can reduce the influence of disturbing factors such as stains and improve recognition accuracy.

[0092] On the other hand, an embodiment of the present invention further discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a cattle facial recognition method based on RGB images and depth images when executing the computer program.

[0093] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0094] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cattle face recognition method based on RGB image and depth image, characterized in that: The following steps are involved: S1: Collect RGB images and corresponding depth images of cow faces at different angles, annotate the RGB images with cow face target frames, and obtain annotated RGB datasets and unannotated RGB datasets; S2: Input the labeled RGB data set into the RetinaFace detection model training, use the trained RetinaFace detection model to perform cow face detection on the unlabeled RGB data set, and synchronously crop the cow face position of the depth image corresponding to each RGB image to obtain the cropped cow face data set; S3: constructing an RGB and depth dual-modal cow face recognition model, wherein the RGB and depth dual-modal cow face recognition model includes: a convolutional feature extraction module, a feature pooling module, an attention refinement module, and a fully connected layer; Among them, the obtained cropped cow face dataset is sent to two convolutional feature extraction modules to extract RGB modal features and deep modal features respectively. The RGB modal features and deep modal features are fused through the feature pooling module, and the features are refined through the attention refinement module to obtain the cow recognition results.

2. The method for cattle facial recognition based on RGB image and depth image according to claim 1, characterized in that: The S1 includes: S11: Use a depth camera to shoot RGBD images of the cow's face from the front and from the left and right sides at 30 degrees, and obtain an RGB image and a corresponding depth image; S12: Screening RGB images and depth images to form a preliminary data set; S13: Perform data enhancement processing on the filtered RGB image and the corresponding depth image simultaneously; S14: Annotate the cow face in the RGB image with a target box to obtain an annotated RGB dataset and an unannotated RGB dataset.

3. The method for cattle facial recognition based on RGB image and depth image according to claim 1, characterized in that: The obtained cropped cow face dataset is sent to two convolutional feature extraction modules to extract RGB modal features and depth modal features respectively, and the RGB modal features and the depth modal features are fused through a feature pooling module, including: S31: Send the RGB image and depth image in the cropped cow face dataset to two VGG-16 convolutional feature extraction modules to extract RGB modal features and depth modal features respectively, and obtain RGB convolution map and depth convolution map; S32: The RGB modal features and the depth modal features are fused through a feature pooling module, and a pooling feature map is established through the feature pooling module.

4. The method for cattle facial recognition based on RGB image and depth image according to claim 3, characterized in that: The feature refinement by the attention refinement module to obtain the cattle recognition result includes: S33: The pooled feature map is used as the input of the attention refinement module, and the pooled features are refined through a fully connected layer with a tanh function activation; S34: Convert the pooled features into an attention map using a convolutional layer with softmax activation; S35: The attention weight is multiplied by the RGB modality feature embedding obtained in S31 and then passed through two fully connected layers, and then a softmax activation is performed to obtain the final recognition score; S36: Find the corresponding cattle number according to the corresponding numerical value of the score to obtain the cattle identification result.

5. The method for cattle facial recognition based on RGB image and depth image according to claim 4, characterized in that: The activation conversion formula for converting pooled features into attention maps using a convolutional layer with softmax activation is as follows: α att =softmax(Conv 1×1 (F sl )); F sl =tanh((W sl ) T F dp ); Among them, Conv 1×1 is a convolutional layer with kernel size of 1×1 and 1 feature map, α att is the obtained attention map, F sl The activated feature tensor, F dp is the correlation feature information between the RGB image and the depth image, W sl Trainable weights for the fully connected layers in the attention refinement module.

6. The method for cattle facial recognition based on RGB image and depth image according to claim 5, characterized in that: The attention weight is multiplied by the RGB modal feature embedding obtained in S31 and then passed through two fully connected layers, and then a softmax activation to obtain the final recognition score, which is expressed as: Among them, W fc1 , W fc2 are the trainable weights of the two fully connected layers in the classifier, F RGB The RGB modality feature embedding output by the convolutional feature extraction module for the RGB channels in the convolutional feature extraction module.

7. The method for cattle facial recognition based on RGB image and depth image according to claim 6, characterized in that: It also includes setting the loss function during the training process. The expressions of each loss function are: Among them, y RGB，c Represents the output vector of the input belonging to a certain category c among M categories, y RGB,c and Depth,c represents the output label of the corresponding input belonging to a category c among M categories, P RGB.c and P Depth.c is the probability score that is the output of the classifier.

8. The method for cattle facial recognition based on RGB image and depth image according to claim 7, characterized in that: The overall loss function is: L Total =L RGB +L D +L attention ; Where, L Total is the overall loss function, L RGB is the RGB image loss function, L D is the depth image loss function, L attention is the attention loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for cattle facial recognition based on RGB images and depth images as described in any one of claims 1 to 8 is implemented.

Images