Pet supervision method and device based on image processing and storage medium

Abstract

The invention is suitable for the technical field of image recognition, and provides a pet supervision method and device based on image processing and a storage medium, and the method comprises the steps: obtaining image data and sound data; wherein the image data comprises a pet image, a user image and an environment image; the sound data comprises pet sound and user sound; segmenting the pet image to obtain a plurality of pet part images; inputting the plurality of pet part images into the plurality of first models to obtain output results of the plurality of first models; and inputting the output results of the plurality of first models, the image data and the sound data into a second model to generate pet supervision information, thereby realizing prediction of aggressive or defensive behaviors of dogs.

Description

Technical Field

[0001] The present application belongs to the field of image recognition technology, and in particular, relates to a pet supervision method, device and storage medium based on image processing. Background Art

[0002] With the development of society, more and more people keep pets, especially dogs. Dogs may react to sudden noises, moving objects, strangers or other animals. And many owners do not tie their dogs when walking them. This will cause dogs to be stimulated by external factors to have aggressive or defensive behaviors, thus posing a safety threat to others. In the prior art, usually the dog is stopped only after it has behaved aggressively or defensively. The prior art lacks a method for predicting whether a dog is about to behave aggressively or defensively. Summary of the invention

[0003] The embodiments of the present application provide a pet supervision method, device and storage medium based on image processing, which can solve the technical problem in the prior art of lacking prediction of a dog's upcoming aggressive or defensive behavior.

[0004] In a first aspect, an embodiment of the present application provides a pet supervision method based on image processing, comprising: Acquire image data and sound data; wherein the image data includes a pet image, a user image, and an environment image; and the sound data includes a pet sound and a user sound; Segmenting the pet image to obtain a plurality of pet part images; Inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models; The output results of the multiple first models, the image data and the sound data are input into the second model to generate pet supervision information.

[0005] In a second aspect, an embodiment of the present application provides a pet monitoring device based on image processing, comprising: An acquisition module, used to acquire image data and sound data; wherein the image data includes pet images, user images and environment images; the sound data includes pet sounds and user sounds; A segmentation module, used for segmenting the pet image to obtain a plurality of pet part images; A first model module, used for inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models; The second model module is used to input the output results of the multiple first models, the image data and the sound data into the second model to generate pet supervision information.

[0006] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image processing-based pet supervision method described in any one of the first aspects above is implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the pet supervision method based on image processing described in any one of the above-mentioned first aspects is implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the pet supervision method based on image processing as described in any one of the first aspects above.

[0009] Compared with the prior art, the embodiments of the present application have the following beneficial effects: acquiring image data and sound data; wherein the image data includes pet images, user images and environment images; the sound data includes pet sounds and user sounds; segmenting the pet images to obtain a number of pet part images; inputting the several pet part images into a plurality of first models to obtain output results of a plurality of first models; inputting the output results of the plurality of first models, the image data and the sound data into a second model to generate pet supervision information, so that the image data can be processed based on the first model and the second model to obtain the pet supervision information, and determining whether the pet has aggressive or defensive behavior based on the pet supervision information, thereby facilitating early response.

[0010] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 is a flowchart of a pet supervision method based on image processing provided by an embodiment of the present application; Figure 2 is a structural schematic diagram of a pet monitoring device based on image processing provided by an embodiment of the present application; Figure 3It is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0014] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0015] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0016] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0017] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0019] Figure 1 A schematic flow chart of a pet supervision method based on image processing provided by the present application is shown.

[0020] S101, acquiring image data and sound data.

[0021] The image data includes pet images, user images and environment images, and the sound data includes pet sounds and user sounds.

[0022] S102, segmenting the pet image to obtain a plurality of pet part images.

[0023] In this application, the pet image to be segmented is input into the trained model, and the model will output the classification result of each pixel. The image is segmented into different parts according to the classification result, and the image of each part is extracted, such as: ear image, eye image, mouth image, tail image, etc.

[0024] S103, inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models.

[0025] Among them, the first model is a lightweight model, which refers to a model that is relatively small in terms of model size, computational complexity, and memory occupancy.

[0026] Among them, each first model corresponds to a first model output result.

[0027] S104, inputting the output results of the multiple first models, the image data and the sound data into the second model to generate pet supervision information.

[0028] The second model may refer to a model that can process and understand multiple types of data, and can process multiple types of data such as text, image, audio, video, etc. at the same time.

[0029] Among them, the pet supervision information can be a summary information.

[0030] Among them, the model specification of the second model is greater than the model specification of the first model.

[0031] In an embodiment of the present application, the acquired pet images are first input into multiple first models to obtain output results of the multiple first models. Thereafter, the output results of the multiple first models, the image data, and the sound data are input into the second model together. By processing multiple types of data, accurate pet supervision information is generated, and based on the pet supervision information, it is determined whether the pet has aggressive or defensive behavior, thereby facilitating advance response.

[0032] In an optional embodiment, the output result includes a first output result, and the step of inputting the plurality of pet body part images into a plurality of first models in S103 to obtain the output results of the plurality of first models includes: A first weight is set for a plurality of pet part images based on the pet type; wherein the pet type is determined based on the pet image; and the plurality of pet part images after the first weight is set are input into a plurality of first models to obtain first output results of the plurality of first models.

[0033] In the embodiment of the present application, different pets have different parts. For example, some dogs have long tails, while others have very short tails. When a dog is nervous, it may clamp its tail. However, for a dog with a very short tail, analyzing its state based on the tail may be inaccurate, and may even lead to errors. For ease of understanding, an example is given here: a trained CNN model is used to identify that this is a Labrador. The dog's ears, eyes, and tail are segmented from the pet image. For a Labrador, the movements of the ears and tail can usually reflect its emotional state well. Therefore, higher weights are set for the ears and tail (for example, the weight of the ears is 0.6, and the weight of the tail is 0.3), and the weight of the eyes is set to 0.1.

[0034] Input to the first model, the weighted ear, eye and tail images are input to the three models respectively: Model A: Analyze ear movements to determine emotions.

[0035] Model B: Analyze eye expressions and judge emotions.

[0036] Model C: Analyze the tail's movements to determine emotions.

[0037] Get the first output result: Model A output: Ear movement indicates the dog is excited.

[0038] Model B output: Eye expression showing the dog is calm.

[0039] Model C output: Tail movement indicates the dog is excited.

[0040] In this way, information from multiple parts can be integrated to more accurately judge the pet's mood and state.

[0041] In an optional embodiment, the output result includes a second output result, and the step of inputting the plurality of pet body part images into a plurality of first models in S103 to obtain the output results of the plurality of first models includes: The user state is determined based on the user image and the user voice, and the environment state is determined based on the environment image; based on the user state and the environment state, the first weight is adjusted to obtain a second weight; a plurality of pet part images after the second weight is set are input into a plurality of first models to obtain second output results of the plurality of first models; the output result is obtained based on the first output result and the second output result.

[0042] In an embodiment of the present application, image recognition technology is used to extract features in user images, analyze the user's facial expressions and body language, and analyze the user's voice based on sound recognition technology to obtain the user's tone, speed and volume. This information can be used to judge the user's emotional state, such as whether it is nervous, excited or calm, so as to determine the user's state. In addition, the environmental image is analyzed, which may include environmental factors such as light, obstacles, and buildings at the location of the pet to obtain the environmental state. This information helps to determine the environmental pressure or comfort that the pet may feel. Then, the first weight is adjusted according to the user state and the environmental state to obtain the second weight. The adjustment of the weight is based on predefined rules or automatically performed by a machine learning algorithm. For example, if a local feature (such as the state of the ear) shows a strong emotional change in a specific environment, while other features do not change much, the weight of the feature needs to be increased.

[0043] Next, the second weight is applied to the judgment of the pet's mood and state. Finally, the output results of the comprehensive model are used to determine the pet's state.

[0044] The output result of the comprehensive model, that is, analyzing the first output result State1 and the second output result State2, and calculating the difference in recognition accuracy between the two, can be expressed by the following formula: ΔS=Accuracy(State2)−Accuracy(State1) If ΔS<0, it means that the second output result is not as good as the first output result, and the reason needs to be analyzed and the weights updated. For example, the importance of certain features (such as ears and eyes) may increase or decrease due to changes in the environment or user status.

[0045] In addition, in addition to analyzing the difference in recognition accuracy between the two, the recognition difference between the two, that is, the difference between the two results, is also analyzed. If the difference is too large, this may indicate that the pet's mood has suddenly changed. The environmental image may be re-analyzed to consider whether there are new changes, such as sudden noises, strangers or other pets, which may cause a sudden change in the pet's mood. At the same time, the user's latest behavior may be analyzed to check whether there are any unusual movements or sounds, which may also be the cause of the pet's mood change. Thereby determining the state of the pet, as well as the environmental state or user state corresponding to the state, is also equivalent to determining the user image, user voice, and environmental image corresponding to the state. The determined user image, user voice, and environmental image corresponding to the state are used as the image data and sound data in S104. The output results determined above and the image data and sound data here are input into the second model to generate pet supervision information.

[0046] If the model identifies that the pet is showing signs of tension, such as ears pulled back and tail tucked, the system may suggest that the owner take some measures to calm the pet, such as providing a quiet environment or performing some relaxing activities.

[0047] For example: User status monitoring: The camera captures your image and the microphone captures your voice. Image analysis shows that your face is tense and voice analysis shows that your tone is sharp.

[0048] Environmental status monitoring: The environmental camera detects that the outdoor light is dim and a large number of items are scattered on the ground in the area. This may distract or make the pet dog feel uneasy.

[0049] Weight adjustment: The model weights for dog emotion analysis are automatically adjusted based on the nervous state and the clutter of the environment. In this scenario, the first weight may refer to the weight of the dog's facial features, such as the weight of the ears, eyes, and tail. Assuming that it is recognized that the dog's ears and tail react most sensitively in a nervous user state, the weight of the ear and tail image recognition is increased. For example, ears clamped back may indicate that the dog is scared or nervous, and a clamped or rapidly wagging tail may also indicate uneasiness. At the same time, if the environment is darker, the weight of the light reflected from the dog's eyes may be increased, because in a low-light environment, the dog's eyes may reflect more light, which can be used as a clue to assess their emotional state.

[0050] Model application: The adjusted weights are applied to the analysis of images of pet dogs. This means that when analyzing the emotions of pet dogs, more attention will be paid to changes in the ears, eyes, and tail.

[0051] Output: The output of the integrated model shows that the dog's ears and tail show signs of tension, and the light reflected from the eyes also shows signs of restlessness. It is recommended that the owner take the dog to a quieter, better-lit area, or perform some relaxing activities such as petting and gentle words to calm the dog.

[0052] In this way, the system can more accurately identify and respond to the pet's emotional state, helping the owner to take timely action.

[0053] In an optional embodiment, S104 inputs the output results of the plurality of first models, the image data and the sound data into the second model to generate pet supervision information, including: Based on the second model, the output results of multiple first models are encoded to obtain a first word vector; feature extraction is performed on the image data to obtain image features; feature extraction and conversion are performed on the sound data to obtain a second word vector; based on the first word vector, the image features and the second word vector, the pet supervision information is generated.

[0054] Wherein, based on the first weight and / or the second weight, an attention weight is set for the combined feature; wherein the combined feature includes the first word vector, the image feature and the second word vector; based on the combined feature with the attention weight set, the pet supervision information is generated.

[0055] In an embodiment of the present application, multiple first models analyze images of different parts of a pet, and the output results are encoded and processed by a second model to generate a first word vector. This vector is a point in a high-dimensional space, representing the emotional state of the pet. In addition, feature extraction is performed on the image data in parallel to obtain image features, and feature extraction and conversion are performed on the sound data to obtain a second word vector. Afterwards, the first word vector, image features, and second word vector are combined to form three rows of parallel features, or they can also be combined into a row of features.

[0056] After that, set the attention weight. Specifically: For each part of the pet, there is a first weight and a second weight corresponding to it. The first weight and the second weight are used to adjust the importance of each feature in the combined feature. For each feature, calculate a weighted score. The formula can be: score=w1*f1+w2*f2, where w1, w1 are the first weight and the second weight, and f1, f2 are the original scores of the corresponding features. Use the softmax function to normalize the weighted scores so that the sum of all scores is 1. Multiply the normalized attention weights by the original features to obtain weighted features. Aggregate the weighted features by summing or averaging to generate the final output feature vector. Input the aggregated weighted features into a classifier, regressor, or decoder to generate pet supervision information.

[0057] The encoding process may be a decoder in a Transformer model. The related technology will not be described in detail here.

[0058] In an embodiment of the present application, image data and sound data are obtained; wherein the image data includes pet images, user images and environment images; the sound data includes pet sounds and user sounds; the pet images are segmented to obtain a number of pet part images; the number of pet part images are input into a plurality of first models to obtain output results of a plurality of first models; the output results of the plurality of first models, the image data and the sound data are input into a second model to generate pet supervision information, so that the image data can be processed based on the first model and the second model to obtain the pet supervision information, and based on the pet supervision information, it is determined whether the pet has aggressive or defensive behavior, so as to facilitate early response.

[0059] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0060] Figure 2 A structural block diagram of a pet monitoring device based on image processing provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0061] Reference Figure 2 , the pet monitoring device based on image processing includes: An acquisition module, used to acquire image data and sound data; wherein the image data includes pet images, user images and environment images; the sound data includes pet sounds and user sounds; A segmentation module, used for segmenting the pet image to obtain a plurality of pet part images; A first model module, used for inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models; The second model module is used to input the output results of the multiple first models, the image data and the sound data into the second model to generate pet supervision information.

[0062] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0063] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a pet supervision method based on image processing is implemented.

[0064] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0065] In one embodiment, a computer device is provided, comprising: a memory, a processor, and an image processing-based pet supervision program stored in the memory and executable on the processor, wherein the image processing-based pet supervision program, when executed by the processor, implements the steps of the image processing-based pet supervision method as described in any of the above embodiments.

[0066] In one embodiment, a computer-readable storage medium is provided, on which a pet supervision program based on image processing is stored. When the pet supervision program based on image processing is executed by a processor, the steps of the pet supervision method based on image processing as described in any of the above embodiments are implemented.

[0067] In one embodiment, a computer program product is provided, which includes an image processing-based pet supervision program, and when the image processing-based pet supervision program is executed by a processor, the steps of the image processing-based pet supervision method described in any of the above embodiments are implemented.

[0068] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0069] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A pet supervision method based on image processing, characterized in that: include: Acquire image data and sound data; wherein the image data includes a pet image, a user image, and an environment image; and the sound data includes a pet sound and a user sound; Segmenting the pet image to obtain a plurality of pet part images; Inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models; The output results of the multiple first models, the image data and the sound data are input into the second model to generate pet supervision information.

2. The pet supervision method based on image processing as claimed in claim 1, characterized in that: The output result includes a first output result; The step of inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models includes: Setting a first weight for a plurality of pet part images based on the pet type; wherein the pet type is determined based on the pet image; The plurality of pet body part images after the first weights are set are input into a plurality of first models to obtain first output results of the plurality of first models.

3. The pet supervision method based on image processing as claimed in claim 2, characterized in that: The output results of the plurality of first models, the image data and the sound data are input into the second model to generate pet supervision information, including: Based on the second model, encoding the output results of the multiple first models to obtain a first word vector; Performing feature extraction on the image data to obtain image features; Extracting and converting features of the sound data to obtain a second word vector; The pet supervision information is generated based on the first word vector, the image feature and the second word vector.

4. The pet supervision method based on image processing as claimed in claim 3, characterized in that: The output result includes a second output result; The step of inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models includes: determining a user state based on the user image and the user voice, and determining an environment state based on the environment image; Adjusting the first weight based on the user state and the environment state to obtain a second weight; Inputting the plurality of pet body part images after the second weights are set into the plurality of first models to obtain second output results of the plurality of first models; The output result is obtained based on the first output result and the second output result.

5. The pet supervision method based on image processing as claimed in claim 4, characterized in that: The generating the pet supervision information based on the first word vector, the image feature and the second word vector includes: Based on the first weight and / or the second weight, setting an attention weight for a combined feature; wherein the combined feature includes the first word vector, the image feature and the second word vector; The pet supervision information is generated based on the combined features with attention weights set.

6. A pet monitoring device based on image processing, characterized in that: include: An acquisition module, used to acquire image data and sound data; wherein the image data includes pet images, user images and environment images; the sound data includes pet sounds and user sounds; A segmentation module, used for segmenting the pet image to obtain a plurality of pet part images; A first model module, used for inputting a plurality of pet body part images into a plurality of first models to obtain output results of the plurality of first models; The second model module is used to input the output results of the multiple first models, the image data and the sound data into the second model to generate pet supervision information.

7. The pet monitoring device based on image processing as claimed in claim 6, characterized in that: The output result includes a first output result; The first model module is used to: Setting a first weight for a plurality of pet part images based on the pet type; wherein the pet type is determined based on the pet image; The plurality of pet body part images after the first weights are set are input into a plurality of first models to obtain first output results of the plurality of first models.

8. The pet monitoring device based on image processing as claimed in claim 7, characterized in that: The second model module is used to: Based on the second model, encoding the output results of the multiple first models to obtain a first word vector; Performing feature extraction on the image data to obtain image features; Extracting and converting features of the sound data to obtain a second word vector; The pet supervision information is generated based on the first word vector, the image feature and the second word vector.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images