Pet image recognition method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a pet image recognition method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: selecting a key frame in a shot scene video, and generating a key frame image by the key frame; sending the key frame image to a backbone network in a pre-trained pet image recognition model for feature extraction to obtain pet image recognition feature data; sending the pet image recognition feature data to a server, so that a branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image to obtain a recognition result of the pet image; and in response to a received pet image recognition result returned by the server, displaying the result on an interface of the user terminal. According to the embodiment, the accuracy of pet image recognition can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of pet image recognition, and in particular to a pet image recognition method, device, electronic device, and computer-readable medium. Background Art

[0002] Pet image recognition is a technology used to identify pets from images. At present, the common method used for pet image recognition is to rely on specific parts of the pet (such as the face) for recognition. Or through general neural network model algorithms such as general detection, tracking and segmentation, the presence of pets can be detected and tracked, as well as pet detection and classification can be performed.

[0003] However, it is found in practice that when the above method is used for pet image recognition, the following technical problems often occur: The shooting angle of the shooting equipment (such as a webcam) and the environmental limitations (such as a home security environment, a pet care environment, etc.) make it difficult to capture images of the pet from a specific perspective (such as a frontal face image), which makes it difficult to effectively identify and classify the facial features of the pet, thereby reducing the accuracy of pet image recognition.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a pet image recognition method, device, electronic device, and computer-readable medium to solve the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a pet image recognition method, which is applied to a shooting device, and the method includes: selecting key frames in a shot scene video, and generating key frame images from the key frames; sending the key frame images to a backbone network in a pre-trained pet image recognition model for feature extraction, and obtaining pet image recognition feature data, wherein the backbone network module of the pet image recognition model is pre-deployed on a user terminal; sending the pet image recognition feature data to a server, so that a branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image, and obtains a recognition result of the pet image; and in response to the pet image recognition result received and returned by the server, displaying the result on an interface of the user terminal.

[0008] Optionally, the above method also includes: performing image quality detection on each frame of video image in the above scene video to generate an image quality detection result set; determining the image quality detection result in the above image quality detection result set that meets preset conditions, and determining the video image corresponding to the image quality detection result that meets the preset conditions as a key frame image.

[0009] Optionally, the method further includes: performing target detection on the scene video to determine a target pet image in the scene video; performing target tracking on the detected target pet image, wherein the identified target pet image is tracked to obtain a tracking identification set; determining a pet image border of the detected target pet image in the scene video image to obtain a pet image detection border set; and performing image quality detection on the video images corresponding to each detected target pet image based on the tracking identification set and the pet image detection border set to generate an image quality detection result set.

[0010] Optionally, the method further includes: in response to determining that there is no image quality detection result satisfying a preset condition in the image quality detection result set, again selecting key frames for the scene video captured by the network camera.

[0011] Optionally, based on the tracking identification set and the pet image detection bounding box set, the video images corresponding to the detected target pet images are subjected to image quality detection to generate an image quality detection result set, including: determining an image clarity set and an image occlusion ratio value set of the video images corresponding to each tracking identification in the tracking identification set according to the pet image detection bounding box set; performing pet position recognition on the video images corresponding to each pet image detection bounding box in the pet image detection bounding box set to generate an image shooting angle information set, wherein each image shooting angle information in the image shooting angle information set corresponds to a pet, and each image shooting angle information includes an observation angle label of the pet relative to the shooting device in the key frame image, and the observation angle label includes at least one of the following: a first angle label, a second angle label or a third angle label; determining an image quality detection result set corresponding to each tracking identification according to the image clarity set, the image occlusion ratio value set and the image shooting angle information set.

[0012] Optionally, the method further includes: determining the image clarity of the video image corresponding to each tracking identifier in the tracking identifier set to obtain an image clarity set; determining the image occlusion degree of the video image corresponding to each tracking identifier in the tracking identifier set to obtain an image occlusion ratio value set.

[0013] Optionally, the pet image recognition model is trained and generated by a preset model generation method; wherein the preset model generation method includes: obtaining a training sample set, wherein the training samples in the training sample set include positive sample images, negative sample image pairs and corresponding positive sample labels and negative sample labels; selecting training samples from the training sample set and inputting them into the backbone network of the initial pet image recognition model to generate positive sample backbone feature maps and negative sample backbone feature maps, wherein the model parameters of the backbone network in the initial pet image recognition model are fixed, and the branch network module in the initial pet image recognition model includes a large cluster branch network and a small cluster branch network, and the branch network module corresponds to a single user; the positive sample backbone feature map is input into the backbone network of the initial pet image recognition model; The image is input into the branch network module of the above-mentioned initial pet image recognition model, and the positive sample prediction label is output, wherein the small clustering branch network in the above-mentioned branch network module is used to cluster the local features of the pet in the positive sample trunk feature map; the negative sample trunk feature map is input into the branch network module of the above-mentioned initial pet image recognition model, and the negative sample prediction label is output, wherein the large clustering branch network in the above-mentioned branch network module is used to cluster the overall features of the pet in the positive sample trunk feature map; a sample loss value is generated according to the obtained positive sample prediction label, the negative sample prediction label, the positive sample label and the negative sample label, and the network parameters in the above-mentioned branch network module are adjusted according to the above-mentioned sample loss value.

[0014] Optionally, the training sample set is generated by the following steps: receiving an image removal operation performed by a user on a pet recognition image in a displayed pet recognition image set; determining the pet recognition image corresponding to the image removal operation as a negative sample image to obtain a negative sample image group, wherein the negative sample image is a negatively correlated image removed by the user; determining the tracking identifier corresponding to each negatively correlated image in the negative sample image group as a negative sample label to obtain a negative sample label group; receiving an image merging operation performed by a user on a pet recognition image in a displayed pet recognition image set; adjusting the tracking identifier corresponding to the image merging operation to the same identifier, and determining each pet recognition image in the pet recognition image set as a positive sample image to obtain a positive sample image group; determining the tracking identifier corresponding to each positive sample image in the positive sample image group as a positive sample label to obtain a positive sample label group; and combining the positive sample image group, the positive sample label group, the negative sample image group and the negative sample label group to obtain a training sample set.

[0015] In a second aspect, some embodiments of the present disclosure provide a pet image recognition device, which includes: a key frame selection unit, configured to select key frames in a captured scene video and generate key frame images from the key frames; a feature extraction unit, configured to send the key frame images to a backbone network in a pre-trained pet image recognition model for feature extraction, and obtain pet image recognition feature data, wherein the backbone network module of the pet image recognition model is pre-deployed on a user terminal; a feature recognition unit, configured to send the pet image recognition feature data to a server, so that a branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image, and obtains a recognition result of the pet image; a recognition result display unit, configured to display the result on an interface of the user terminal in response to the pet image recognition result received and returned by the server.

[0016] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0017] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0018] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the pet image recognition method of some embodiments of the present disclosure, the accuracy of pet image recognition can be improved. Specifically, the reason for the reduced accuracy of pet image recognition is that the shooting angle and environmental restrictions (for example, home security environment) of the shooting device (such as a webcam) make it difficult to capture images of pets from a specific perspective (for example, a frontal face image), thereby making it difficult to effectively identify and classify the facial features of the pet. Based on this, the pet image recognition method of some embodiments of the present disclosure first selects key frames for the scene video shot by the shooting device to generate key frame images. By extracting the key frame images, it can be used to select images with appropriate image perspectives for image recognition. Then, the key frame images are sent to the backbone network in the pre-trained pet image recognition model for feature extraction to obtain pet image recognition feature data. Among them, the backbone network module of the above pet image recognition model is pre-deployed on the user terminal. Here, by deploying the backbone network of the pet image recognition model in advance on the user terminal, it can be used to extract features through the user terminal, thereby alleviating the computing resource pressure on the server. At the same time, because the backbone network deployed for the user terminals of each user is the same, it can be easily deployed and updated. Then, the above-mentioned pet image recognition feature data is sent to the server side, so that the branch network module corresponding to the above-mentioned pet image recognition model in the above-mentioned server side can perform feature recognition of the above-mentioned pet image to obtain the recognition result of the above-mentioned pet image. Here, the recognition of the pet image is realized by the branch network module set on the server side. Finally, in response to the above-mentioned pet image recognition result returned by the above-mentioned server side, the result is displayed on the interface of the above-mentioned user terminal. Therefore, the above-mentioned implementation method can first be used to select appropriate video images as key frame images, so as to avoid different perspectives of the pet in the image during the recognition process, which affects the recognition accuracy. Furthermore, the facial features of the pet can be effectively recognized and classified, and the accuracy of pet image recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0020] Figure 1 is a flow chart of some embodiments of the pet image recognition method according to the present disclosure; Figure 2 is a schematic diagram for image privacy protection according to some embodiments of the pet image recognition method disclosed herein; Figure 3is a schematic diagram of multi-feature fusion by a branch network module in some embodiments of the pet image recognition method disclosed herein; Figure 4 are flow charts of other embodiments of the pet image recognition method according to the present disclosure; Figure 5 is a schematic diagram of the structure of some embodiments of the pet image recognition device according to the present disclosure; Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0023] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0024] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0026] With regard to the collection, storage, and use of user personal information (such as user videos and user pet information) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] Figure 1The process 100 of some embodiments of the pet image recognition method according to the present disclosure is shown. The pet image recognition method is applied to a shooting device (such as a network camera), and includes the following steps: Step 101, selecting key frames in the captured scene video, and generating key frame images from the key frames.

[0029] In some embodiments, the execution subject of the pet image recognition method can select key frames in the captured scene video and generate key frame images from the key frames. Among them, a video image with object movement in the scene video can be determined by a motion detection algorithm to serve as a key frame image.

[0030] As an example, the motion detection algorithm may include but is not limited to at least one of the following: a frame difference method, an optical flow method, a background subtraction method, and the like.

[0031] It should be noted that the above wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0032] Optionally, the execution subject may further include a privacy protection module. The privacy protection module may perform the following steps before sending the key frame image to the pre-trained pet image recognition model: The key frame image is subjected to a person blurring process to remove the person image information therein. The person position region in the key frame image can be located by a person detection algorithm. Then, the person position region can be subjected to a blurring process by a video local blurring method. Thus, the person image information therein is removed. Thus, the purpose of protecting user privacy is achieved.

[0033] As an example, the person detection algorithm may include but is not limited to at least one of the following: a YOLO (You Only LookOnce) algorithm, a CNN (Convolutional Neural Networks) algorithm, and the like.

[0034] As another example, the function of the privacy protection module can be as follows: Figure 2 Through character blurring, the character in the key frame image can be blurred to shield the user's privacy and avoid leaking user information during the subsequent processing of the key frame image.

[0035] Step 102, sending the key frame image to the backbone network in the pre-trained pet image recognition model to extract features and obtain pet image recognition feature data.

[0036] In some embodiments, the execution subject may send the key frame image to a backbone network in a pre-trained pet image recognition model to extract features and obtain pet image recognition feature data. The backbone network module of the pet image recognition model is pre-deployed on a user terminal. The pet image recognition feature data may include at least one recognition feature map corresponding to a pet.

[0037] Here, the backbone network may be a preset network structure, for example, including but not limited to at least one of the following: a backbone network of a VGG (Visual Geometry Group Network, convolutional neural network) model and a Google Net (deep neural network) model. The output of the backbone network module corresponds to the input of the branch network module.

[0038] In practice, the backbone network modules deployed for user terminals of different users may be the same, which can improve deployment efficiency and facilitate unified upgrading of the backbone network.

[0039] Optionally, the feature extraction can be used to extract multiple part features of the pet, including but not limited to at least one of the following: facial features, trunk features, and limb features, so as to perform multi-feature fusion recognition.

[0040] Optionally, the pet image recognition model is generated by training using a preset model generation method. The preset model generation method includes: The first step is to obtain a training sample set. The training samples in the training sample set include positive sample images, negative sample image pairs, and corresponding positive sample labels and negative sample labels. The positive sample image can be an image of a pet. The positive sample label can be a bright label corresponding to the pet in the image. The negative sample image can be an image that does not include a pet. The negative sample label can be any label.

[0041] In the second step, a training sample is selected from the training sample set and input into the backbone network of the initial pet image recognition model to generate a positive sample backbone feature map and a negative sample backbone feature map. The model parameters of the backbone network in the initial pet image recognition model are fixed. The branch network module in the initial pet image recognition model may include a large cluster branch network and a small cluster branch network. The branch network module corresponds to a single user. Here, the parameters of the branch network modules of different users can be trained based on the user's personal pet images, so there are differences in the model parameters.

[0042] As an example, the backbone network may include convolutional layers, residual blocks, identity connections, global average pooling, and fully connected layers. A convolutional layer may include multiple convolutional groups. Each convolutional group may include computational processes such as convolution operations, normalization operations, and activations. Each residual block typically contains two convolutional layers, each followed by a batch normalization layer and a ReLU activation function. The input is directly added to the output to form a skip connection. The identity connection allows the input to directly skip the output of some layers and then be added to the output of these layers.

[0043] In the third step, the positive sample trunk feature map is input into the branch network module of the initial pet image recognition model, and the positive sample prediction label is output. The small cluster branch network in the branch network module can be used to cluster the pet local features in the positive sample trunk feature map. Here, the small cluster branch network can be a network layer constructed by a clustering algorithm.

[0044] Here, considering that pets have changeable postures in the home environment, only some body parts (such as trunk and limbs) may be photographed. Therefore, in the training process of the model, the selection of the number of clusters can be based on the data distribution characteristics, or the optimal number of clusters can be determined by the elbow method. In the small cluster branch network layer, the average result can be obtained by multiple clustering. In addition, the local features of the pet in the positive sample trunk feature map may include but are not limited to at least one of the following: pet facial features, pet trunk features, pet back features, etc. Thus, the local features are clustered by clustering to form multiple small cluster centers to increase the inter-class distance between small clusters. Thus, the degree of distinction between different pets can be improved by improving the control of the inter-class distance of the local features of the pets. For example, cats and dogs with similar pet colors, body shapes, postures, etc. are prone to recognition errors. Then, the clustering results can be dynamically adjusted to adapt to the changes in the pet's posture and appearance through the different distance features reflected in the image due to the different structures of the pets themselves. Thus, the problems of pet posture changeability and viewing angle limitation are solved, and the recognition accuracy is further improved.

[0045] As an example, the clustering algorithm may include but is not limited to at least one of the following: K-means clustering algorithm or DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. Figure 3 As shown in the figure, the branch network module can perform multiple feature fusion on the input features and output the positive sample prediction label.

[0046] In the fourth step, the negative sample trunk feature map is input into the branch network module of the initial pet image recognition model, and the negative sample prediction label is output. The large cluster branch network in the branch network module can be used to cluster the overall features of the pet in the positive sample trunk feature map. Here, the large cluster branch network can also be constructed by the clustering algorithm.

[0047] Specifically, the large cluster branch network can use the tracking mark corresponding to the image to combine multiple small clusters from the same pet into a large cluster to characterize an individual pet. Therefore, through the diverse feature description, it can be used to reduce the intra-class distance within the large cluster and improve the overall recognition stability of the pet.

[0048] In addition, the branch network module may also include a fully connected layer. The branch network module may superimpose the features output by the small cluster branch network and the large cluster branch network as the overall features and input them into the fully connected layer for pet recognition, and output the positive sample trunk feature map and the negative sample trunk feature map of the positive and negative sample images, respectively, and output the positive sample prediction label and the negative sample prediction label.

[0049] The fifth step is to generate a sample loss value based on the obtained positive sample prediction label, the obtained negative sample prediction label, the obtained positive sample label and the obtained negative sample label, and adjust the network parameters in the branch network module based on the obtained sample loss value. The sample loss value can be generated by a preset loss function.

[0050] As an example, the loss function may include but is not limited to at least one of the following: a mean square error loss function, a cross entropy loss function, a multi-category cross entropy loss function, and the like.

[0051] Optionally, it is determined that the training of the initial pet image recognition model is not completed (for example, the sample loss value is greater than a preset loss threshold), and the network parameters in the branch network module are adjusted according to the sample loss value.

[0052] Optionally, the above training sample set can be generated by the following steps: The first step is to receive a user's image removal operation for a pet recognition image in the displayed pet recognition image set, wherein the image removal operation may be an operation by the user to delete one or more pet recognition images.

[0053] In the second step, the pet recognition image corresponding to the above image removal operation is determined as a negative sample image to obtain a negative sample image group, wherein the negative sample image can be a negatively correlated image removed by the user.

[0054] In practice, in order to further improve the model's ability to recognize user pets for different customers' different pets, by using images manually deleted by users as negative samples, the model can be provided with erroneous image features, reducing the probability of false detection and improving the generalization ability of the model.

[0055] In the third step, the tracking identifier corresponding to each negatively correlated image in the negative sample image group is determined as a negative sample label to obtain a negative sample label group.

[0056] The fourth step is to receive an image merging operation of the user on the pet recognition images in the displayed pet recognition image set, wherein the image merging operation may be an operation of the user merging at least two pet recognition images into an image corresponding to the same pet.

[0057] In the fifth step, the tracking identifiers corresponding to the above-mentioned image merging operation are adjusted to the same identifier, and each pet recognition image in the pet recognition image set is determined as a positive sample image to obtain a positive sample image group.

[0058] In practice, in addition to the pet recognition images that the user has not removed, by determining the merged images as positive samples, the model can be helped to better classify images belonging to the same category during the training process. Thus, the learning ability of the model is enhanced, so that the model can be trained according to the user's personalized features, quickly adapt to user needs, and improve the recognition accuracy of the model.

[0059] In the sixth step, the tracking identifier corresponding to each positive sample image in the positive sample image group is determined as a positive sample label to obtain a positive sample label group.

[0060] In the seventh step, the positive sample image group, positive sample label group, negative sample image group and negative sample label group are combined to form a training sample set.

[0061] In practice, the above model training method can achieve efficient, accurate and personalized pet recognition by introducing a user-specific dual-branch Head and combining it with a personalized training strategy. In addition, user feedback and data accumulation can be used to achieve continuous evolution and performance improvement of the model.

[0062] Step 103, sending the pet image recognition feature data to the server, so that the branch network module corresponding to the pet image recognition model in the server can perform feature recognition of the pet image to obtain the recognition result of the pet image.

[0063] In some embodiments, the execution subject may send the pet image recognition feature data to the server, so that the branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image to obtain the recognition result of the pet image. The pet image recognition result may include information such as a pet image recognition box and a pet image recognition name. Here, the pet image recognition name may correspond to a name or logo preset by the user.

[0064] In practice, the branch network modules of the server corresponding to each user terminal may be the same or different. Specifically, the difference may refer to: the branch network module is obtained after personalized model training for each user. Here, the personalized model training may be trained using the user's personal pet images.

[0065] Step 104, in response to the pet image recognition result returned by the server, the result is displayed on the interface of the user terminal.

[0066] In some embodiments, the execution subject may display the result on the interface of the user terminal in response to the pet image recognition result received from the server. The key frame image and the pet image recognition box and pet image recognition name in the pet image recognition result at the marked position in the key frame image may be displayed on the interface of the user terminal.

[0067] As an example, the system architecture of the execution entity may include: a webcam, a cloud service, a local computing center (i.e., the execution entity mentioned above), a client, and other components. Specifically, the execution entity mentioned above may obtain a scene video captured by the webcam. Then, feature extraction is performed on the scene video, and pet image recognition feature data is obtained. Afterwards, the pet image recognition feature data may be sent to a server (such as the cloud service) for feature recognition of the pet image based on a branch network module corresponding to the user end stored in the server, to obtain a recognition result of the pet image. Finally, the recognition result may be sent to the client to display the result on the interface.

[0068] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the pet image recognition method of some embodiments of the present disclosure, the accuracy of pet image recognition can be improved. Specifically, the reason for the reduced accuracy of pet image recognition is that the shooting angle and environmental restrictions (for example, home security environment) of the shooting device (such as a webcam) make it difficult to capture images of pets from a specific perspective (for example, a frontal face image), thereby making it difficult to effectively identify and classify the facial features of the pet. Based on this, the pet image recognition method of some embodiments of the present disclosure first selects key frames for the scene video shot by the shooting device to generate key frame images. By extracting the key frame images, it can be used to select images with appropriate image perspectives for image recognition. Then, the key frame images are sent to the backbone network in the pre-trained pet image recognition model for feature extraction to obtain pet image recognition feature data. Among them, the backbone network module of the above pet image recognition model is pre-deployed on the user terminal. Here, by deploying the backbone network of the pet image recognition model in advance on the user terminal, it can be used to extract features through the user terminal, thereby alleviating the computing resource pressure on the server. At the same time, because the backbone network deployed for the user terminals of each user is the same, it can be easily deployed and updated. Then, the above-mentioned pet image recognition feature data is sent to the server side, so that the branch network module corresponding to the above-mentioned pet image recognition model in the above-mentioned server side can perform feature recognition of the above-mentioned pet image to obtain the recognition result of the above-mentioned pet image. Here, the recognition of the pet image is realized by the branch network module set on the server side. Finally, in response to the above-mentioned pet image recognition result returned by the above-mentioned server side, the result is displayed on the interface of the above-mentioned user terminal. Therefore, the above-mentioned implementation method can first be used to select appropriate video images as key frame images, so as to avoid different perspectives of the pet in the image during the recognition process, which affects the recognition accuracy. Furthermore, the facial features of the pet can be effectively recognized and classified, and the accuracy of pet image recognition can be improved.

[0069] Continue to refer Figure 4 , shows a process 400 of some embodiments of the pet image recognition method according to the present disclosure. The process 400 of the pet image recognition method includes the following steps: Step 401: Perform image quality detection on each frame of video image in the scene video to generate an image quality detection result set.

[0070] In some embodiments, the execution subject of the pet image recognition method can perform image quality detection on each frame of the video image in the scene video to generate an image quality detection result set. In particular, the image quality detection can be performed on each video image in the scene video using a pre-trained quality detection model.

[0071] As an example, the quality detection model may include but is not limited to at least one of the following: a CNN (Convolutional Neural Networks) model or a Transformer converter model. Here, the training samples of the quality detection model may be positive and negative sample pairs consisting of pre-selected high-quality samples and low-quality samples.

[0072] Specifically, the training of the quality detection model can be carried out simultaneously by supervised learning and unsupervised learning. Supervised learning can be to input the training sample pairs into the model for supervised training. The unsupervised learning method can be: using the deep feature map output by the quality detection model in the supervised learning process to determine the image similarity of each area in the feature map. Here, the true value of the similarity between image areas corresponding to the same tracking mark is 1. The true value of the similarity between image areas with different tracking marks is 0. Therefore, comparative learning can be carried out by determining the image similarity of each area. In order to train the model on unlabeled data, learn the feature space structure in the feature map, and enhance the generalization ability of the model.

[0073] In some optional implementations of some embodiments, the above execution entity may include the following steps: In the first step, target detection is performed on the scene video to determine the target pet image in the scene video. In particular, a target detection algorithm (such as YOLO, SSD) can be used to detect the pet target in the video stream in real time.

[0074] As an example, the target detection algorithm may include but is not limited to at least one of the following: a YOLO (You Only Look Once-Version) target detection model and an SSD (Single Shot MultiBox Detector) model.

[0075] The second step is to track the detected target pet image, wherein the identified target pet image is tracked to obtain a tracking identification set. The detected target pet image can be tracked by a multi-target tracking algorithm, and a corresponding tracking identification can be assigned.

[0076] As an example, the multi-target tracking algorithm may include but is not limited to at least one of the following: a DeepSORT multi-target tracking algorithm, a nearest neighbor algorithm, and the like.

[0077] The third step is to determine the pet image border of the detected target pet image in the above scene video image to obtain a pet image detection border set. The pet image border of the detected target pet image in the above scene video image can be determined by a preset segmentation algorithm.

[0078] As an example, the above-mentioned segmentation algorithm may include but is not limited to at least one of the following: Mask R-CNN (Mask Region-based Convolutional Neural Network, image instance segmentation) model, SAM (Segment Anything Model, image segmentation) model.

[0079] The fourth step is to perform image quality detection on the video images corresponding to each detected target pet image based on the tracking identification set and the pet image detection bounding box set to generate an image quality detection result set.

[0080] Optionally, the execution subject performs image quality detection on the video images corresponding to each detected target pet image based on the tracking identification set and the pet image detection bounding box set to generate an image quality detection result set, which may include the following steps: In the first step, based on the pet image detection frame set, an image definition set and an image occlusion ratio value set of the video image corresponding to each tracking mark in the tracking mark set are determined.

[0081] Optionally, the above execution entity may include the following steps: Step 1, determine the image clarity of the video image corresponding to each tracking mark in the above tracking mark set in the video image, and obtain the image clarity set. Among them, for the video image corresponding to each tracking mark, the corresponding image clarity can be determined by the following steps: First, the image gradient of the video image can be determined by the Sobel operator. A clear image has a higher gradient value. Then, the high-frequency component of the video image can be determined by Fourier transform. Here, the high-frequency component of a clear image is often richer. Finally, the gradient value and the value of the high-frequency component can be weighted to obtain the image clarity.

[0082] Step 2, determine the image occlusion degree of the video image corresponding to each tracking identifier in the above tracking identifier set in the video image, and obtain an image occlusion ratio value set. Among them, for the pet image detection border in the video image corresponding to each tracking identifier, the corresponding image occlusion ratio value can be determined by the following steps: First, the integrity of the pet image detection border can be determined by the above target detection algorithm. It is used to detect whether the pet in the image is partially obscured. Then, if it is obscured, the ratio of the number of pixels in the obscured area to the number of pixels in the unobstructed area can be determined as the image occlusion ratio value.

[0083] In the second step, the pet position recognition is performed on the video images corresponding to each of the above pet image detection bounding boxes in the above pet image detection bounding box set to generate an image shooting angle information set. Among them, each image shooting angle information in the above image shooting angle information set may correspond to a pet. Each image shooting angle information may include an observation angle label of the pet relative to the above network camera in the key frame image. The observation angle label may include but is not limited to at least one of the following: a front perspective label, a side perspective label or a back perspective label. Here, the pet position recognition can be performed on the video images corresponding to each of the above pet image detection bounding boxes in the above pet image detection bounding box set through a preset pet posture estimation model to generate an image shooting angle information set.

[0084] The fourth step is to determine the image quality detection result set corresponding to each tracking mark according to the above image clarity set, the above image occlusion ratio value set and the above image shooting angle information set. For each video image, if the image clarity of the corresponding pet image detection border area is greater than the preset clarity threshold, the image occlusion ratio value is less than the preset occlusion ratio threshold, and the image shooting angle information includes a preset observation angle label, the pass mark indicating that the image quality detection has passed is determined as the image quality detection result.

[0085] In practice, the above-mentioned embodiments comprehensively utilize detection, tracking, segmentation, quality assessment and user feedback to construct a complete pet individual recognition system, which can be used to efficiently and accurately perform pet image recognition.

[0086] Step 402: determine an image quality detection result that meets a preset condition in the image quality detection result set, and determine a video image corresponding to the image quality detection result that meets the preset condition as a key frame image.

[0087] In some embodiments, the execution subject may determine, in response to determining that there is an image quality detection result that satisfies a preset condition in the image quality detection result set, a video image corresponding to the image quality detection result that satisfies the preset condition in the image quality detection result set as a key frame image. The preset condition may be that the image quality detection result includes a pass mark indicating that the detection has passed.

[0088] Optionally, the above execution entity may further perform the following steps: In response to determining that there is no image quality detection result that meets the preset conditions in the image quality detection result set, key frame selection is performed again on the scene video captured by the network camera. If each image quality detection result does not meet the preset conditions, key frame selection can be performed again from the latest scene video captured by the network camera.

[0089] Step 403, sending the key frame image to the backbone network in the pre-trained pet image recognition model to extract features and obtain pet image recognition feature data.

[0090] Step 404, sending the pet image recognition feature data to the server, so that the branch network module corresponding to the pet image recognition model in the server can perform feature recognition of the pet image to obtain the recognition result of the pet image.

[0091] Step 405: In response to the pet image recognition result received from the server, the result is displayed on the interface of the user terminal.

[0092] In some embodiments, the specific implementation of the above steps 403 to 405 and the technical effects brought about can be referred to Figure 1 The corresponding steps 101 to 103 in the embodiments are not described in detail here.

[0093] from Figure 4 It can be seen that Figure 4 Compared with the description of some corresponding embodiments, Figure 4 The corresponding pet image recognition process 400 in some embodiments embodies the steps of image quality detection for each frame of video image. Therefore, the scheme described in these embodiments can enhance the integrity and technical depth of the scheme by refining the implementation of the quality judgment module. Combined with supervised and unsupervised learning methods, a high-precision quality assessment model is constructed by using tracking ID and other methods. The accuracy of key frame selection is improved, providing a reliable data basis for subsequent pet individual identification.

[0094] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a pet image recognition device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0095] like Figure 5As shown, the pet image recognition generation device 500 of some embodiments includes: a key frame selection unit 501, a feature extraction unit 502, a feature recognition unit 503, and a recognition result display unit 504. Among them, the key frame selection unit 501 is configured to select key frames in the captured scene video and generate key frame images from the key frames; the feature extraction unit 502 is configured to send the key frame images to the backbone network in the pre-trained pet image recognition model for feature extraction to obtain pet image recognition feature data, wherein the backbone network module of the pet image recognition model is pre-deployed in the user terminal; the feature recognition unit 503 is configured to send the pet image recognition feature data to the server, so that the branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image to obtain the recognition result of the pet image; the recognition result display unit 504 is configured to display the result on the interface of the user terminal in response to the pet image recognition result returned by the server.

[0096] It is understood that the units described in the device 500 are similar to those described in the reference Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0097] Reference below Figure 6 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0098] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage device 608 to a random access memory 603. Various programs and data required for the operation of the electronic device 600 are also stored in the random access memory 603. The processing device 601, the read-only memory 602, and the random access memory 603 are connected to each other via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0099] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0100] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a read-only memory 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0101] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0102] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0103] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: selects key frames in the captured scene video and generates key frame images from the key frames; sends the key frame images to the backbone network in the pre-trained pet image recognition model for feature extraction to obtain pet image recognition feature data, wherein the backbone network module of the pet image recognition model is pre-deployed in the user terminal; sends the pet image recognition feature data to the server, so that the branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image to obtain the recognition result of the pet image; and displays the result on the interface of the user terminal in response to the pet image recognition result received from the server.

[0104] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a key frame selection unit, a feature extraction unit, a feature recognition unit, and a recognition result display unit. The names of these units do not constitute limitations on the units themselves in certain circumstances, for example, the key frame selection unit may also be described as a "unit for performing key frame selection on a scene video captured by a network camera to generate a key frame image."

[0107] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0108] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A pet image recognition method, applied to a photographing device, characterized in that: The method comprises: Selecting key frames in the captured scene video, and generating key frame images from the key frames; Sending the key frame image to a backbone network in a pre-trained pet image recognition model to extract features and obtain pet image recognition feature data, wherein the backbone network module of the pet image recognition model is pre-deployed on a user terminal; The pet image recognition feature data is sent to a server, so that a branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image to obtain a recognition result of the pet image; In response to the pet image recognition result received and returned by the server, the result is displayed on the interface of the user terminal.

2. The method according to claim 1, characterized in that Before sending the key frame image to the pre-trained pet image recognition model, perform the following steps: The key frame image is subjected to a person blurring process to remove the person image information therein.

3. The method according to claim 1, characterized in that The method further comprises: performing image quality detection on each frame of video image in the scene video to generate an image quality detection result set; An image quality detection result that meets a preset condition in the image quality detection result set is determined, and a video image corresponding to the image quality detection result that meets the preset condition is determined as a key frame image.

4. The method according to claim 1 or 3, characterized in that: The method further comprises: Performing target detection on the scene video to determine a target pet image in the scene video; Performing target tracking on the detected target pet image, wherein the identified target pet image is tracked to obtain a tracking identification set; Determine a pet image border of the detected target pet image in the scene video image to obtain a pet image detection border set; Based on the tracking identification set and the pet image detection frame set, image quality detection is performed on the video images corresponding to each detected target pet image to generate an image quality detection result set.

5. The method according to claim 3, characterized in that: The method further comprises: In response to determining that there is no image quality detection result satisfying a preset condition in the image quality detection result set, key frame selection is performed again on the scene video captured by the network camera.

6. The method according to claim 4, characterized in that The performing of image quality detection on the video images corresponding to each detected target pet image based on the tracking identification set and the pet image detection frame set to generate an image quality detection result set includes: Determine, according to the pet image detection frame set, an image definition set and an image occlusion ratio value set of the video image corresponding to each tracking identifier in the tracking identifier set; Performing pet position recognition on the video images corresponding to each of the pet image detection frames in the pet image detection frame set to generate an image shooting angle information set, wherein each image shooting angle information in the image shooting angle information set corresponds to a pet, and each image shooting angle information includes an observation angle label of the pet relative to the shooting device in the key frame image, and the observation angle label includes at least one of the following: a first perspective label, a second perspective label, or a third perspective label; An image quality detection result set corresponding to each tracking identifier is determined according to the image clarity set, the image occlusion ratio value set and the image shooting angle information set.

7. The method according to claim 6, characterized in that The method further comprises: Determine the image definition of the video image corresponding to each tracking identifier in the tracking identifier set to obtain an image definition set; The image occlusion degree of the video image corresponding to each tracking identifier in the tracking identifier set is determined to obtain an image occlusion ratio value set.

8. The method according to claim 1, characterized in that The pet image recognition model is generated by training through a preset model generation method; wherein the preset model generation method comprises: Acquire a training sample set, wherein the training samples in the training sample set include positive sample images, negative sample image pairs and corresponding positive sample labels and negative sample labels; Selecting training samples from the training sample set and inputting them into the backbone network of the initial pet image recognition model to generate a positive sample backbone feature map and a negative sample backbone feature map, wherein the model parameters of the backbone network in the initial pet image recognition model are fixed, and the branch network module in the initial pet image recognition model includes a large cluster branch network and a small cluster branch network, and the branch network module corresponds to a single user; Inputting the positive sample trunk feature map into the branch network module of the initial pet image recognition model, and outputting the positive sample prediction label, wherein the small clustering branch network in the branch network module is used to cluster the pet local features in the positive sample trunk feature map; Inputting the negative sample trunk feature map into the branch network module of the initial pet image recognition model, and outputting the negative sample prediction label, wherein the large clustering branch network in the branch network module is used to cluster the overall features of the pet in the positive sample trunk feature map; Generate a sample loss value according to the obtained positive sample prediction label, the negative sample prediction label, the positive sample label and the negative sample label; The network parameters in the branch network module are adjusted according to the sample loss value.

9. The method according to claim 8, characterized in that The training sample set is generated by the following steps: receiving an image removal operation of a user on a pet identification image in the displayed pet identification image set; Determine the pet recognition image corresponding to the image removal operation as a negative sample image, and obtain a negative sample image group, wherein the negative sample image is a negatively correlated image removed by the user; Determine the tracking identifier corresponding to each negatively correlated image in the negative sample image group as a negative sample label to obtain a negative sample label group; receiving an image merging operation of a user on a pet recognition image in a displayed pet recognition image set; The tracking identifiers corresponding to the image merging operation are adjusted to be the same identifier, and each pet identification image in the pet identification image set is determined as a positive sample image to obtain a positive sample image group; Determine the tracking identifier corresponding to each positive sample image in the positive sample image group as a positive sample label to obtain a positive sample label group; The positive sample image group, the positive sample label group, the negative sample image group and the negative sample label group are combined to form a training sample set.

10. The method according to claim 1, characterized in that The feature extraction is used to extract various part features of the pet, including at least one of the following: facial features, trunk features and limb features, so as to perform multi-feature fusion recognition.

11. A pet image recognition device, comprising: A key frame selection unit is configured to select key frames in the captured scene video and generate key frame images from the key frames; A feature extraction unit is configured to send the key frame image to a backbone network in a pre-trained pet image recognition model to extract features and obtain pet image recognition feature data, wherein the backbone network module of the pet image recognition model is pre-deployed on a user terminal; A feature recognition unit is configured to send the pet image recognition feature data to a server, so that a branch network module corresponding to the pet image recognition model in the server performs feature recognition of the pet image to obtain a recognition result of the pet image; The recognition result display unit is configured to display the result on the interface of the user terminal in response to the pet image recognition result received and returned by the server.

12. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 10.

13. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images