Pet state identification method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a pet state recognition method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: determining a signal source position according to at least one positioning signal in response to the received at least one positioning signal sent by pet identification equipment; monitoring audios and videos are collected in real time according to the position of the signal source; determining a pet type according to the signal source position and a monitoring video included in the monitoring audio and video; in response to the fact that the pet type and the audio description information meet a first condition, determining pet state description information according to the signal source position and the monitoring audio and video; and in response to the fact that the pet type and the audio description information do not meet the first condition, determining pet state description information according to the signal source position and the monitoring video included in the monitoring audio and video. According to the embodiment, accurate and effective pet state identification is realized.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a pet status recognition method, device, electronic device, and computer-readable medium. Background Art

[0002] Pets are a general term for animals that are raised and cared for by humans. Since pets have attributes such as companionship, more and more people are starting to raise pets as family members. However, since common pets such as cats and dogs are small in size and highly flexible in movement, it is difficult to effectively monitor the real-time status of pets in indoor spaces with large areas and complex spatial environments. According to relevant reports, there are often cases of pet deaths caused by abnormal behaviors such as accidental ingestion. Therefore, how to effectively and accurately identify the status of pets has become an urgent problem to be solved.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a pet status recognition method, device, electronic device, and computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a pet status recognition method, which is applied to a pet recognition device, comprising: in response to receiving at least one positioning signal sent by the above-mentioned pet recognition device, determining the signal source position according to the above-mentioned at least one positioning signal; according to the above-mentioned signal source position, collecting monitoring audio and video in real time; determining the pet type according to the above-mentioned signal source position and the monitoring video included in the above-mentioned monitoring audio and video; in response to the above-mentioned pet type and audio description information satisfying a first condition, determining the pet status description information according to the above-mentioned signal source position and the above-mentioned monitoring audio and video, wherein the above-mentioned audio description information is an audio description of the monitoring audio included in the above-mentioned monitoring audio and video, and the above-mentioned pet status description information includes: pet behavior description information and pet health description information; in response to the above-mentioned pet type and the above-mentioned audio description information not satisfying the above-mentioned first condition, determining the above-mentioned pet status description information according to the above-mentioned signal source position and the monitoring video included in the above-mentioned monitoring audio and video.

[0007] Optionally, the above-mentioned determining the pet type according to the above-mentioned signal source position and the surveillance video included in the above-mentioned surveillance audio and video includes: determining the pet type according to the above-mentioned signal source position, the surveillance video included in the above-mentioned surveillance audio and video and the pet identification corresponding to the positioning signal, and binding the above-mentioned pet identification with the identified pet target.

[0008] Optionally, determining the signal source position based on the at least one positioning signal includes: for each of the at least one positioning signal, performing the following processing steps: determining a target information set corresponding to the positioning signal, wherein the target information in the target information set includes: a signal time difference, a first coordinate and a second coordinate, the signal time difference representing a time difference between a signal receiver corresponding to the first coordinate and a signal receiver corresponding to the second coordinate receiving the positioning signal; determining a candidate signal source position based on the target information set; and performing signal source position correction based on the obtained candidate signal source position set to generate the signal source position.

[0009] Optionally, the pet behavior description information includes: pet behavior type and pet behavior duration, wherein the pet behavior duration represents the duration that the pet maintains the behavior corresponding to the pet behavior type, and the pet health description information includes: pet health; and the method further includes: cumulatively updating the pet health according to the pet behavior type and the pet behavior duration to obtain an updated pet health; in response to the updated pet health being within the first health interval, sending abnormal health prompt information to the target client; in response to the pet behavior type being an abnormal behavior type and the pet behavior duration being greater than the abnormal behavior duration threshold corresponding to the pet behavior type, sending abnormal pet behavior prompt information to the target client.

[0010] Optionally, the above-mentioned determination of the pet type according to the above-mentioned signal source position and the monitoring video included in the above-mentioned monitoring audio and video includes: performing shallow image feature extraction on the above-mentioned monitoring video to obtain a shallow image feature sequence; determining the signal source type according to the above-mentioned shallow image feature sequence, wherein the above-mentioned signal source type includes: a living type and a non-living type, and the above-mentioned living type includes: a pet living type and a non-pet living type; in response to the above-mentioned signal source type being a pet living type, determining a region of interest according to the above-mentioned signal source position, wherein the above-mentioned region of interest includes: at least one superimposed region of interest; according to the above-mentioned at least one superimposed region of interest, performing feature windowing on each shallow image feature in the above-mentioned shallow image feature sequence to generate a windowed shallow image feature to obtain a windowed shallow image feature sequence; performing deep image feature extraction on the above-mentioned windowed shallow image feature sequence to generate a deep image feature to obtain a deep image feature sequence; determining the above-mentioned pet type according to the above-mentioned deep image feature sequence.

[0011] Optionally, the audio description information includes: audio source description information and audio quality description information, the audio source description information characterizing the sound source contained in the monitoring audio, and the audio description information is generated through the following steps: extracting audio features from the monitoring audio to generate audio features; generating the audio source description information through an audio source recognition model and the audio features; generating the audio quality description information based on an audio quality scoring model, the audio features and the audio source description information.

[0012] Optionally, the pet status description information is determined based on the signal source position and the monitoring audio and video, including: performing video feature extraction on the monitoring video to generate video features, wherein the video feature extraction model shares the shallow image feature extraction model and the deep image feature extraction model included in the pet recognition model; performing audio feature extraction on the monitoring audio to generate audio features, wherein the audio feature extraction model is the audio feature extraction model in the audio description information generation model; performing feature fusion on the video features and the audio features to generate fusion features; generating the pet status description information based on the fusion features and a pet status description information predictor, wherein the pet status description information predictor includes: a pet behavior type predictor and a pet health predictor.

[0013] Optionally, the method further includes: determining whether video uploading is allowed; in response to permission, partially blurring the surveillance video to obtain a partially blurred surveillance video; compressing the partially blurred surveillance video to generate a compressed video; and storing the compressed video locally and in the cloud respectively.

[0014] Optionally, the above-mentioned local video blurring of the above-mentioned surveillance video to obtain the locally blurred surveillance video includes: determining the regional confidence of each of the above-mentioned at least one superimposed region of interest, wherein the regional confidence represents the confidence that the corresponding superimposed region of interest contains a pet; based on the region size of the superimposed region of interest and the regional confidence corresponding to the superimposed region of interest, screening out the superimposed region of interest that meets the second condition from the above-mentioned at least one superimposed region of interest as the target region of interest; based on the above-mentioned target region of interest, inversely selecting and cropping each shallow image feature in the above-mentioned shallow image feature sequence to generate a cropped shallow image feature to obtain a cropped shallow image feature sequence; based on the above-mentioned cropped shallow image feature sequence and the non-pet living body positioning model, determining the position of the non-pet living body type object in the above-mentioned surveillance video to obtain the non-pet living body position; and locally blurring the video at the above-mentioned non-pet living body position in the above-mentioned surveillance video to obtain the locally blurred surveillance video.

[0015] In a second aspect, some embodiments of the present disclosure provide a pet identification equipment, including: a pet locator, wherein the pet locator includes: at least two ultra-wideband signal transmitters, and the pet locator is fixed to the pet's body surface in a wearable form; a signal acquisition device, wherein the signal acquisition device includes: at least three ultra-wideband signal receivers, a video acquisition device and an audio acquisition device, and the at least three ultra-wideband signal receivers, the video acquisition device and the audio acquisition device face the same acquisition surface; one or more processors; a storage device, on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0016] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0017] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the pet status recognition method of some embodiments of the present disclosure, effective and accurate pet status recognition is achieved. Specifically, the reason why the pet status recognition cannot be performed effectively and accurately is that common pets such as cats and dogs have the characteristics of small size and high flexibility of movement, and it is difficult to effectively monitor and identify the real-time status of pets in indoor spaces with complex spatial environments (such as indoor spaces with furniture). Based on this, the present disclosure first responds to receiving at least one positioning signal sent by the above-mentioned pet recognition device, and determines the signal source position according to the above-mentioned at least one positioning signal, so as to obtain the real-time position of the signal source. Secondly, according to the above-mentioned signal source position, real-time monitoring audio and video are collected, and then, according to the above-mentioned signal source position and the monitoring video included in the above-mentioned monitoring audio and video, the pet type is determined. In this way, an accurate pet type is obtained. Further, in response to the above-mentioned pet type and audio description information satisfying the first condition, the pet status description information is determined according to the above-mentioned signal source position and the above-mentioned monitoring audio and video, wherein the above-mentioned audio description information is an audio description of the monitoring audio included in the above-mentioned monitoring audio and video, and the above-mentioned pet status description information includes: pet behavior description information and pet health description information. Finally, in response to the above-mentioned pet type and the above-mentioned audio description information not satisfying the above-mentioned first condition, the above-mentioned pet status description information is determined according to the above-mentioned signal source position and the monitoring video included in the above-mentioned monitoring audio and video. In practice, the calls of some pets are relatively small or the noise is relatively large during collection, resulting in poor quality of the obtained monitoring audio. At this time, using the monitoring audio as one of the bases for determining the pet status description information will increase the noise and affect the accuracy of the obtained pet status description information. Therefore, when the monitoring audio is poor, the pet behavior is comprehensively determined only in combination with the video and the signal source position. In this way, the pet status is effectively and accurately identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0019] Figure 1 is a flow chart of some embodiments of the pet status identification method according to the present disclosure; Figure 2 A front view of a signal collection device included in the pet identification equipment; Figure 3 is a schematic diagram of a pet locator included in the pet identification equipment; Figure 4 This is a schematic diagram of the single pet target recognition process; Figure 5It is a schematic diagram of the recognition process for multiple pet targets; Figure 6 is another schematic diagram of the recognition process for multiple pet targets; Figure 7 is another flow chart of some embodiments of the pet status identification method according to the present disclosure; Figure 8 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0022] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0026] refer to Figure 1 , shows a process 100 of some embodiments of the pet status identification method according to the present disclosure. The pet status identification method comprises the following steps: Step 101, in response to receiving at least one positioning signal sent by a pet identification device, determining a signal source position according to the at least one positioning signal.

[0027] In some embodiments, the execution subject (e.g., a computing device) of the pet status identification method may determine the signal source location according to the at least one positioning signal received from the pet identification device in response to receiving at least one positioning signal sent by the pet identification device. Among them, at least one positioning signal may be emitted by a pet locator included in the pet identification device. The signal source location may represent the location of the pet's locator. In practice, since the pet locator is fixed to the pet's body surface in a wearable form, the signal source location may represent the pet's location. The above-mentioned execution subject may receive the above-mentioned at least one positioning signal through the ultra-wideband signal receiver included in the pet identification device.

[0028] As an example, the execution subject can determine the position of the signal source according to the incident angle of the positioning signal received by the ultra-wideband signal receiver and the signal propagation time. Specifically, since the speed of light is known and the receiver positions of at least three ultra-wideband signal receivers are known, the position of the signal source from the ultra-wideband signal receiver can be obtained according to the signal propagation time and the speed of light. Through the incident angle of the positioning signal and the ultra-wideband signal receiver, the position of the signal source can be calculated.

[0029] As another example, the signal acquisition device may include: at least three ultra-wideband signal receivers, a video acquisition device, and an audio acquisition device, wherein the at least three ultra-wideband signal receivers, the video acquisition device, and the audio acquisition device face the same acquisition plane. Figure 2 The pet identification equipment shown in the figure includes a front view of a signal collection device, wherein the signal collection device includes: at least three ultra-wideband signal receivers 1 (specifically, Figure 2 Two groups of four ultra-wideband signal receivers 1), a video acquisition device 1 and an audio acquisition device 3 are shown, and the at least three ultra-wideband signal receivers 1, the video acquisition device 2 and the audio acquisition device 3 are facing the same acquisition surface. Specifically, the video acquisition device includes: a main camera 5 and a sub-camera 6. In practice, the main camera 5 is a wide-angle camera. The sub-camera 6 is an infrared camera. The audio acquisition device 3 is a directional microphone. In addition, the signal acquisition device also includes a rotating table 4. The signal acquisition device can be installed on the top surface of the wall through the rotating table 4, and the rotation of the rotating table 4 can be controlled to control the change of the acquisition surface of the signal acquisition device. In practice, the rotating table 4 supports horizontal rotation and 30-degree pitch rotation.

[0030] As yet another example, see Figure 3 The pet identification equipment shown is a schematic diagram of a pet locator, wherein the pet locator can be a pet collar 7. The pet collar 7 includes: at least two ultra-wideband signal transmitters 8. Figure 3The two ultra-wideband signal transmitters 8 shown are arranged in a symmetrical structure. For another example, for three ultra-wideband signal transmitters 7, an ultra-wideband signal transmitter 7 can be arranged every 120 degrees on the pet collar 7. Specifically, the ultra-wideband signal transmitter 7 has the characteristics of low power consumption, so it is powered by a button battery. In addition, the pet locator corresponds to the pet identity, that is, at least two ultra-wideband signal transmitters correspond to the same pet identity. Among them, the pet identity is the unique identity of the pet. In addition, the ultra-wideband signal transmitter corresponds to a device identifier. For example, when the pet locator includes two ultra-wideband signal transmitters, that is, ultra-wideband signal transmitter A and ultra-wideband signal transmitter B. Among them, ultra-wideband signal transmitter A corresponds to device identifier A. Ultra-wideband signal transmitter B corresponds to device identifier B. The device identifier corresponding to each ultra-wideband signal transmitter will be encoded in the frame structure of the ultra-wideband signal. Optionally, the pet locator can also be fixed on pet clothes, pet tags, etc. For example, magic sticky can be used to fix the pet locator on pet clothes. For another example, the pet locator may be fixed in a pet tag by an embedded setting.

[0031] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as a single software or software module. No specific limitation is made here. In practice, the above-mentioned computing device can be a processor included in the signal acquisition device, for example, a CPU (Central Processing Unit).

[0032] Step 102, collecting monitoring audio and video in real time according to the signal source location.

[0033] In some embodiments, the execution subject may collect monitoring audio and video in real time according to the signal source location. The monitoring audio and video may be a video including video images and audio at the signal source location. In practice, the execution subject may control a video acquisition device and an audio acquisition device to collect video images and audio at the signal source location as the monitoring audio and video.

[0034] Step 103, determining the pet type according to the signal source location and the surveillance video included in the surveillance audio and video.

[0035] In some embodiments, the above-mentioned execution subject can determine the pet type according to the signal source location and the monitoring video included in the monitoring audio and video. Among them, the pet type can characterize the specific type of the corresponding pet at the signal source location. For example, the pet type may include: Maine Coon cat type, Cow cat type, Calico cat type, Teddy dog ​​type, Golden Retriever dog type. In practice, the above-mentioned execution subject can perform picture recognition on the picture at the signal source location in the monitoring video to determine the pet type.

[0036] In some optional implementations of some embodiments, the execution subject determines the pet type according to the signal source location and the surveillance video included in the surveillance audio and video, including: The pet type is determined based on the signal source location, the surveillance video included in the surveillance audio and video, and the pet identity corresponding to the positioning signal, and the pet identity is bound to the identified pet target.

[0037] In practice, the execution subject can determine the area of ​​interest in the surveillance video according to the signal source location. The area of ​​interest can be an area containing pets. Then, pet identification is performed in the area of ​​interest to identify the pet target and the pet type corresponding to the pet target. Then, the execution subject can assign the pet identity corresponding to the positioning signal to the pet target to achieve binding and automatic calibration.

[0038] By combining pet identification with visual recognition algorithms, not only can pet targets be automatically calibrated, replacing manual calibration methods and improving calibration efficiency, but it can also significantly improve the accuracy and efficiency of pet recognition. In addition, the model is continuously trained and optimized by combining training samples obtained through automatic calibration to further improve the recognition accuracy and efficiency of the model, while achieving the effect of exclusive customization of the model.

[0039] As an example, see Figure 4 The schematic diagram of the single pet target recognition process shown in the figure, wherein pet A (for example, puppy A) transmits a positioning signal to the signal acquisition device through the pet locator worn by the pet. At this time, the signal acquisition device can collect surveillance video. Among them, the area of ​​interest in the surveillance video is the area containing pet A. Then, pet recognition can be performed in the area of ​​interest to identify pet target A, and the pet identity corresponding to pet A is assigned to the identified pet target A to achieve binding and automatic calibration.

[0040] As yet another example, see Figure 5The schematic diagram of the multi-pet target recognition process shown in the figure, wherein pet A1 (e.g., puppy A1) and pet A2 (e.g., puppy A2) transmit positioning signals to the signal acquisition device through the pet locator they wear. At this time, the signal acquisition device can collect surveillance videos. Among them, the surveillance video can be two different surveillance videos for pet A1 and pet A2, or it can be a surveillance video containing pet A1 and pet A2. For the former, the two surveillance videos can each include an area of ​​interest, corresponding to pet A1 and pet A2 respectively. For the latter, the surveillance video can include two areas of interest, corresponding to pet A1 and pet A2, in which case, there can be an area intersection between the two areas of interest. Then, pet recognition can be performed in the area of ​​interest to identify the pet target A1, and assign the pet identity corresponding to pet A1 to the identified pet target A1, and identify the pet target A2, and assign the pet identity corresponding to pet target A2 to the identified pet target A1, so as to achieve binding and automatic calibration.

[0041] As yet another example, see Figure 6 The schematic diagram of the multi-pet target recognition process shown in the figure, wherein pet A1 (e.g., puppy A1) and pet B1 (e.g., kitten B1) transmit positioning signals to the signal acquisition device through the pet locator they wear. At this time, the signal acquisition device can collect surveillance videos. Among them, the surveillance video can be two different surveillance videos for pet A1 and pet B1, or it can be a surveillance video containing pet A1 and pet B1. For the former, the two surveillance videos can each include an area of ​​interest, corresponding to pet A1 and pet B1 respectively. For the latter, the surveillance video can include two areas of interest, corresponding to pet A1 and pet B1, in which case, there can be an area intersection between the two areas of interest. Then, pet recognition can be performed in the area of ​​interest to identify pet target A1, and assign the pet identity corresponding to pet A1 to the identified pet target A1, and identify pet target A2, and assign the pet identity corresponding to pet target B1 to the identified pet target B1, so as to achieve binding and automatic calibration.

[0042] Step 104, in response to the pet type and the audio description information satisfying the first condition, determine the pet status description information according to the signal source location and the monitored audio and video.

[0043] In some embodiments, in response to the pet type and the audio description information satisfying the first condition, the above-mentioned execution subject can determine the pet status description information according to the signal source location and the monitoring audio and video. Among them, the audio description information is an audio description of the monitoring audio included in the above-mentioned monitoring audio and video. The pet status description information includes: pet behavior description information and pet health description information. The above-mentioned first condition is: the average pet volume of the pet represented by the pet type is greater than or equal to the preset pet volume, and the audio quality of the monitoring audio represented by the audio description information is greater than the preset audio quality. In practice, some small animals have a low volume or do not make any sound when they make sounds, so it is impossible to effectively combine the corresponding monitoring audio to generate pet status description information. In addition, the collected monitoring audio may include sounds from multiple sources. When there are multiple sources, there may be confusion between the sounds, affecting the audio quality.

[0044] As an example, the execution subject can obtain the pet status description information by using a multimodal model, taking the picture of the signal source position in the monitoring video and the audio emitted by the signal source position in the monitoring audio as input. For example, the multimodal model can be a Video-LLaMA2 model.

[0045] Optionally, the audio description information may include: audio source description information and audio quality description information. The audio source description information characterizes the sound source contained in the monitoring audio. The audio quality description information may characterize the audio quality of the monitoring audio. The pet behavior description information is used to describe the pet behavior. The pet health description information is used to describe the current health status of the pet.

[0046] Optionally, the audio description information is generated by the following steps: The first step is to extract audio features from the above monitoring audio to generate audio features.

[0047] In practice, surveillance audio has the characteristics of time series, so the audio feature extraction model uses a recurrent neural network model as the backbone network.

[0048] The second step is to generate the above audio source description information through the audio source recognition model and the above audio features.

[0049] In practice, the audio source recognition model uses a fully connected layer as a predictor to predict the number of sound sources contained in the monitored audio as audio source description information.

[0050] The third step is to generate the audio quality description information according to the audio quality scoring model, the audio features and the audio source description information.

[0051] The audio feature extraction model, the audio source identification model and the audio quality scoring model are included in the audio description information generation model. In practice, the audio quality scoring model takes the audio features and the audio description information as input to obtain an audio quality score for the monitored audio. Specifically, a regression model can be used as the audio quality scoring model to obtain the audio quality description information.

[0052] Step 105, in response to the pet type and the audio description information not satisfying the first condition, determining the pet status description information according to the signal source location and the monitoring video included in the monitoring audio and video.

[0053] In some embodiments, in response to the pet type and the audio description information not satisfying the first condition, the execution entity may determine the pet status description information based on the signal source location and the surveillance video included in the surveillance audio and video.

[0054] As an example, the execution subject may obtain the pet status description information by using a single-modality (video modality) model, such as a YOLO (You Only Look Once) model, by monitoring the image of the signal source position in the video as input.

[0055] Optionally, the present disclosure may adopt a "cloud-edge architecture", that is, the algorithms corresponding to steps 101 to 105 involved in the present disclosure may be deployed to a local device (e.g., a camera or a local server). The model involved in the present disclosure may be trained and updated on a cloud device (e.g., a cluster server), as well as stored and analyzed for big data. The "cloud-edge architecture" may fully utilize the computing power of local devices at the edge, while efficiently utilizing cloud devices for rapid model training and updating.

[0056] In addition, it should be noted that for operations such as collection, storage and analysis of user-related information, before performing the corresponding operations, the obligation to inform the corresponding subjects of the user-related information has been fulfilled, and the authorization and consent of the corresponding subjects of the user-related information have been obtained in advance.

[0057] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the pet status recognition method of some embodiments of the present disclosure, effective and accurate pet status recognition is achieved. Specifically, the reason why the pet status recognition cannot be performed effectively and accurately is that common pets such as cats and dogs have the characteristics of small size and high flexibility of movement, and it is difficult to effectively monitor and identify the real-time status of pets in indoor spaces with complex spatial environments (such as indoor spaces with furniture). Based on this, the present disclosure first responds to receiving at least one positioning signal sent by the above-mentioned pet recognition device, and determines the signal source position according to the above-mentioned at least one positioning signal, so as to obtain the real-time position of the signal source. Secondly, according to the above-mentioned signal source position, real-time monitoring audio and video are collected, and then, according to the above-mentioned signal source position and the monitoring video included in the above-mentioned monitoring audio and video, the pet type is determined. In this way, an accurate pet type is obtained. Further, in response to the above-mentioned pet type and audio description information satisfying the first condition, the pet status description information is determined according to the above-mentioned signal source position and the above-mentioned monitoring audio and video, wherein the above-mentioned audio description information is an audio description of the monitoring audio included in the above-mentioned monitoring audio and video, and the above-mentioned pet status description information includes: pet behavior description information and pet health description information. Finally, in response to the above-mentioned pet type and the above-mentioned audio description information not satisfying the above-mentioned first condition, the above-mentioned pet status description information is determined according to the above-mentioned signal source position and the monitoring video included in the above-mentioned monitoring audio and video. In practice, the calls of some pets are relatively small or the noise is relatively large during collection, resulting in poor quality of the obtained monitoring audio. At this time, using the monitoring audio as one of the bases for determining the pet status description information will increase the noise and affect the accuracy of the obtained pet status description information. Therefore, when the monitoring audio is poor, the pet behavior is comprehensively determined only in combination with the video and the signal source position. In this way, the pet status is effectively and accurately identified.

[0058] Further references Figure 7 , which shows a process 700 of another embodiment of a pet status identification method. The process 700 of the pet status identification method includes the following steps: Step 701, in response to receiving at least one positioning signal sent by a pet identification device, determining a signal source position according to the at least one positioning signal.

[0059] In some embodiments, the execution subject (e.g., computing device) of the pet status identification method may determine the signal source location according to the at least one positioning signal in response to receiving at least one positioning signal sent by the pet identification device, and may include the following steps: In the first step, for each positioning signal of the at least one positioning signal, the following processing steps are performed: The first sub-step is to determine the target information set corresponding to the above positioning signal.

[0060] The target information in the target information set includes: a signal time difference, a first coordinate, and a second coordinate. The signal time difference represents a time difference between when a signal receiver corresponding to the first coordinate and a signal receiver corresponding to the second coordinate receive the positioning signal.

[0061] As an example, the at least three ultra-wideband signal receivers may include: ultra-wideband signal receiver A, ultra-wideband signal receiver B, and ultra-wideband signal receiver C. The first coordinate may be the position coordinate of ultra-wideband signal receiver A. The second coordinate may be the position coordinate of ultra-wideband signal receiver B. The signal time difference may be the time difference between ultra-wideband signal receiver A and ultra-wideband signal receiver B receiving the positioning signal.

[0062] The second sub-step is to determine the candidate signal source positions based on the above target information set.

[0063] As an example, at least one positioning signal includes: positioning signal A1 and positioning signal A2. At least two ultra-wideband signal transmitters include: ultra-wideband signal transmitter B1 and ultra-wideband signal transmitter B2. At least three ultra-wideband signal receivers include: ultra-wideband signal receiver ultra-wideband signal receiver C1, ultra-wideband signal receiver C2 and ultra-wideband signal receiver C3. Among them, positioning signal A1 is sent by ultra-wideband signal transmitter B1. Positioning signal A2 is sent by ultra-wideband signal transmitter B2. Since the speed of light and the receiver position coordinates of ultra-wideband signal receivers C1, ultra-wideband signal receivers C2 and ultra-wideband signal receivers C3 are known. Therefore, for the transmitter position coordinates of any ultra-wideband signal transmitter, three sets of equations can be constructed. Taking ultra-wideband signal receivers C1 and ultra-wideband signal receivers C2 as examples, the transmitter position coordinates (first coordinates) of ultra-wideband signal receiver C1 and the transmitter position coordinates (second coordinates) of ultra-wideband signal receiver C1 are known, so the position difference between the first coordinate and the signal source coordinates - the position difference between the second coordinate and the signal source coordinates = speed of light × signal time difference. By constructing at least three sets of equations, the candidate signal source locations can be solved.

[0064] In the second step, the signal source position is corrected according to the obtained set of candidate signal source positions to generate the above signal source position.

[0065] In practice, since at least one positioning signal is received, multiple candidate signal source positions are obtained. In order to eliminate positioning errors, the position mean of the obtained set of candidate signal source positions can be used as the signal source position.

[0066] Step 702, collecting monitoring audio and video in real time according to the signal source location.

[0067] In some embodiments, the specific implementation of step 702 and the technical effects thereof can be referred to in Figure 1 The corresponding step 102 in the embodiment will not be described in detail here.

[0068] Step 703, determining the pet type according to the signal source location and the surveillance video included in the surveillance audio and video.

[0069] In some embodiments, the execution subject determines the pet type according to the signal source location and the surveillance video included in the surveillance audio and video, which may include the following steps: The first step is to extract shallow image features from the above surveillance video to obtain a shallow image feature sequence.

[0070] Among them, since the surveillance video is a collection of multiple surveillance images, the above-mentioned execution subject can extract shallow image features from the surveillance images in the surveillance video to obtain a shallow image feature sequence. Shallow image features are image features for surveillance images under a high receptive field.

[0071] In practice, the above-mentioned execution entity can perform shallow image feature extraction on the above-mentioned monitoring video through a shallow image feature extraction model to obtain a shallow image feature sequence.

[0072] As an example, the shallow image feature extraction model includes 18 serially connected convolutional layers. Specifically, the shallow image feature extraction model includes: convolutional layer A1, convolutional layer A2, convolutional layer A3, convolutional layer A4, convolutional layer A5, convolutional layer A6, convolutional layer A7, convolutional layer A8, convolutional layer A9, convolutional layer A10, convolutional layer A11, convolutional layer A12, convolutional layer A13, convolutional layer A14, convolutional layer A15, convolutional layer A16, convolutional layer A17, convolutional layer A18. Among them, the output of convolutional layer A1 is the input of convolutional layer A2 and the input of convolutional layer A4. Specifically, the output of convolutional layer Ai is the input of convolutional layer Ai+1 and the input of convolutional layer Ai+3. Among them, i=2n+1, n is 0 or a positive integer, and n≤7.

[0073] The second step is to determine the signal source type based on the above shallow image feature sequence.

[0074] The signal source types include: living body type and non-living body type. The living body types include: pet living body type and non-pet living body type. The non-pet living body type may represent a person. In practice, the execution subject may determine the signal source type according to the shallow image feature sequence through a signal source type classifier.

[0075] As an example, the signal source type classifier may include: 4 serially connected fully connected layers. The signal source type classifier is connected to the above-mentioned shallow image feature extraction model, that is, the signal source type classifier takes the output of the shallow image feature extraction model as input. The output size of the last fully connected layer among the 4 serially connected fully connected layers is 1×3, corresponding to the confidence of the pet live type, the non-pet live type, and the non-live type, respectively. In the third step, in response to the signal source type being a living pet type, a region of interest is determined according to the signal source position.

[0076] Wherein, the above-mentioned region of interest includes: at least one superimposed region of interest.

[0077] In practice, first, the signal source position is the three-dimensional coordinate of the signal source in the earth coordinate system (XYZ coordinate system). Therefore, it is necessary to map it through coordinate transformation to obtain the coordinates of the signal source position in the image coordinate system (UV coordinate system). Among them, since the main camera 5 and the auxiliary camera 6 included in the video acquisition device are calibrated before use, the signal source position can be mapped to the image coordinate system through coordinate transformation to obtain the coordinate representation of the signal source position in the image coordinate system as the target signal source position. Secondly, the above-mentioned execution subject can construct K interest frames with the target signal source position as the center. Among them, K is greater than or equal to 3. Since the K interest frames are all centered on the target signal source position, the K interest frames are in a superposition relationship, that is, there is a regional intersection between any two interest frames, which can be understood as at least one superimposed interest region.

[0078] In the fourth step, according to the at least one superimposed region of interest, feature windowing is performed on each shallow image feature in the shallow image feature sequence to generate a windowed shallow image feature and obtain a windowed shallow image feature sequence.

[0079] In practice, for shallow image features, the shallow image features can be divided into K layers, each layer corresponding to an overlay region of interest. K is the number of overlay regions of interest in at least one overlay region of interest. The feature values ​​contained in the overlay regions of interest in each layer remain unchanged, and the feature values ​​outside the overlay regions of interest in the layer are set to 0. Then the K processed layers are used as the windowed shallow image features corresponding to the shallow image features.

[0080] The fifth step is to extract deep image features from the above windowed shallow image feature sequence to generate deep image features and obtain a deep image feature sequence.

[0081] The deep image features may be image features for monitoring images under low receptive fields. In practice, the execution subject may extract deep image features from the windowed shallow image feature sequence through a deep image feature extraction model to generate deep image features and obtain a deep image feature sequence.

[0082] As an example, the deep image feature extraction model uses a feature pyramid network. Specifically, the deep image feature extraction model uses a downsampling network composed of five convolutional layers to extract deep image features from the above windowed shallow image feature sequence to generate deep image features.

[0083] Step 6: Determine the pet type based on the deep image feature sequence.

[0084] In practice, the execution subject can determine the pet type according to the deep image feature sequence through a pet type classifier. The pet type classifier is a multi-classifier. The shallow image feature extraction model, the signal source type classifier, the deep image feature extraction model and the pet type classifier are included in the pet recognition model.

[0085] Optionally, after determining the pet type, the pet identity identifier can be associated with the deep graphic features to generate annotated training samples. In addition, missed recognition and / or wrong recognition cases within a fixed time period can be collected as negative (training) samples to achieve the purpose of sample optimization. At the same time, by continuously generating training samples through pet images collected under different environments, lighting conditions, and shooting angles, the purpose of continuously enriching the training sample library can be achieved, on this basis, the robustness and accuracy of the pet type classifier can be improved. Moreover, it can also achieve the purpose of personalized model.

[0086] Step 704, in response to the pet type and the audio description information satisfying the first condition, the pet status description information is determined according to the signal source location and the monitored audio and video.

[0087] In some embodiments, the execution subject determines the pet status description information according to the signal source location and the monitored audio and video in response to the pet type and the audio description information satisfying the first condition, which may include the following steps: The first step is to extract video features from the above surveillance video to generate video features.

[0088] The video feature extraction model shares the shallow image feature extraction model and the deep image feature extraction model included in the pet recognition model. In addition, in order to avoid repeated feature extraction, the deep image feature sequence can be directly used as the video feature.

[0089] The second step is to extract audio features from the above monitoring audio to generate audio features.

[0090] The audio feature extraction model is the audio feature extraction model in the audio description information generation model. In addition, in order to avoid repeated feature extraction, the audio features obtained by extracting audio features from the monitoring audio through the audio feature extraction model can be directly used as the audio features obtained in the second step.

[0091] The third step is to fuse the above video features and the above audio features to generate fused features.

[0092] In practice, the execution entity can fuse the video features and the audio features through a fusion model to generate fusion features. Specifically, in order to fuse the video features and the audio features, the fusion model disclosed herein adopts a cross model based on an attention mechanism as a fusion model to achieve the fusion of video features and audio features. Specifically, the fusion model includes: a cross model A and a feedforward network A based on an attention mechanism, and a cross model B and a feedforward network B based on an attention mechanism. Among them, the cross model A based on the attention mechanism and the feedforward network A are connected in series to extract features from the video features. The cross model B based on the attention mechanism and the feedforward network B are connected in series to extract features from the audio features. Then, the extracted video features and the extracted audio features are feature spliced ​​as fusion features.

[0093] The fourth step is to generate the above-mentioned pet status description information based on the above-mentioned fusion features and the pet status description information predictor.

[0094] The pet state description information predictor includes: a pet behavior type predictor and a pet health predictor. The pet behavior type predictor and the pet health predictor are both multi-classifiers. The pet behavior type predictor is used to classify pet behavior types. For example, pet eating behavior type, pet sleeping behavior type, pet walking behavior type.

[0095] Step 705, in response to the pet type and the audio description information not satisfying the first condition, determining the pet status description information according to the signal source location and the monitoring video included in the monitoring audio and video.

[0096] In some embodiments, the above-mentioned execution subject can determine the pet status description information according to the signal source location and the monitoring video included in the monitoring audio and video in response to the pet type and audio description information not satisfying the first condition. In practice, since the use of monitoring audio is not involved at this time, the video features can be directly used as fusion features and input into the pet status description information predictor to generate the above-mentioned pet status description information. At this time, it is considered that the feature dimensions of the fusion features containing only video features and the fusion features containing video features and audio features are different. Therefore, before the fusion features are input into the pet status description information predictor, the feature dimensions need to be increased to keep consistent with the feature dimensions of the fusion features containing video features and audio features.

[0097] Step 706, cumulatively updating the pet health according to the pet behavior type and the pet behavior duration, to obtain an updated pet health.

[0098] In some embodiments, the above-mentioned execution entity can cumulatively update the pet health according to the pet behavior type and the pet behavior duration to obtain an updated pet health.

[0099] In practice, each pet behavior type corresponds to a corresponding basic score, the basic score is accumulated with a bias score according to the duration of the pet behavior, and the pet health is updated to obtain an updated pet health.

[0100] Step 707: In response to the updated pet health being within the first health interval, sending health abnormality prompt information to the target client.

[0101] In some embodiments, the above-mentioned execution entity may send health abnormality prompt information to the target client in response to the updated pet health being within the first health interval. The first health interval represents that the pet is in a score interval where the health is abnormal. In practice, the health abnormality prompt information may be "Hello XX, the health of your pet is XX, which is low. Please check your pet in time." The target client may be a client bound to an account corresponding to the pet identity identifier corresponding to the pet locator.

[0102] Step 708: In response to the pet behavior type being an abnormal behavior type and the pet behavior duration being greater than the abnormal behavior duration threshold corresponding to the pet behavior type, pet behavior abnormality prompt information is sent to the target client.

[0103] In some embodiments, the execution subject may send pet behavior abnormality prompt information to the target client in response to the pet behavior type being an abnormal behavior type and the pet behavior duration being greater than the abnormal behavior duration threshold corresponding to the pet behavior type.

[0104] In practice, the abnormal pet behavior prompt message can be "Hello XX, your pet's current behavior is abnormal and it lasts for a long time. Please pay attention to your pet in time."

[0105] Optionally, the above method further includes: The first step is to determine whether video uploading is allowed.

[0106] In practice, in order to relieve local storage pressure and ensure the effectiveness of video backtracking, it is necessary to upload the video. However, since the surveillance video is a surveillance video of the user's residence and other areas, in order to avoid user privacy leakage, the video upload behavior needs to be confirmed before the video is uploaded. Specifically, a video upload permission confirmation can be sent to the target client bound to the user. The surveillance video can be uploaded only when the user permits video upload. When the user does not permit video upload, only local storage of the surveillance video is allowed.

[0107] The second step is to locally blur the surveillance video in response to permission to obtain the locally blurred surveillance video.

[0108] In practice, the above-mentioned execution entity can perform Gaussian blur on the privacy area in the surveillance video to obtain the surveillance video after local blur.

[0109] In some optional implementations of some embodiments, the execution subject partially blurs the surveillance video to obtain the partially blurred surveillance video, including: The first sub-step is to determine the region confidence of each of the at least one superimposed region of interest.

[0110] Among them, the regional confidence represents the confidence that the corresponding superimposed region of interest contains a pet.

[0111] In practice, when the pet type classifier outputs the pet type, it will obtain the confidence for different pet types. Therefore, the regional confidence corresponding to each superimposed region of interest can be determined by the pet type classifier.

[0112] In the second sub-step, according to the region size of the superimposed region of interest and the region confidence corresponding to the superimposed region of interest, a superimposed region of interest satisfying a second condition is screened out from the at least one superimposed region of interest as a target region of interest.

[0113] The second condition is that the size of the region superimposed with the region of interest is relatively the largest and the corresponding region confidence is relatively the largest.

[0114] In the third sub-step, according to the target region of interest, each shallow image feature in the shallow image feature sequence is inversely selected and cropped to generate cropped shallow image features, thereby obtaining a cropped shallow image feature sequence.

[0115] In practice, the execution subject may set the feature values ​​in the target region of interest in the shallow image features to 0 to obtain the cropped shallow image features.

[0116] The fourth sub-step is to determine the position of the non-pet living object in the surveillance video based on the cropped shallow image feature sequence and the non-pet living object positioning model to obtain the non-pet living object position.

[0117] Among them, the non-pet living body position represents the position of the person in the video. In practice, the non-pet living body positioning model can adopt the Lite-Uet model, which can achieve accurate position positioning and the lightweight model can reduce the pressure of data processing.

[0118] The fifth sub-step is to locally blur the video at the non-pet living position in the surveillance video to obtain a locally blurred surveillance video.

[0119] In practice, the execution subject may use a fuzzy algorithm such as Gaussian blur to locally blur the video at the non-pet living position in the surveillance video to obtain a locally blurred surveillance video.

[0120] The third step is to compress the above-mentioned locally blurred surveillance video to generate a compressed video.

[0121] In practice, the execution entity may perform video compression on the locally blurred surveillance video using an algorithm such as H.265 to generate a compressed video.

[0122] The fourth step is to store the compressed video locally and in the cloud.

[0123] In practice, the above-mentioned execution entity can upload the compressed video to a cloud server and store it locally.

[0124] Optionally, during the data transmission process, the transmitted data is encrypted.

[0125] For example, when uploading a compressed video to the cloud, the compressed video is encrypted. For another example, when a pet identification device sends a positioning signal, the positioning signal is encrypted.

[0126] Through data encryption transmission and blurring of people included in the surveillance video, the user's privacy data is protected, thereby improving user trust and satisfaction.

[0127] from Figure 7 It can be seen that Figure 1 Compared with the description of some corresponding embodiments, the present disclosure first refines the method of determining the location of the signal source and improves the positioning accuracy. Secondly, by combining the pet behavior description information, the pet's health and abnormal behavior are monitored in real time, and timely warnings are issued. Then, on the premise of obtaining the user's upload permission confirmation, in order to alleviate the local storage pressure and ensure the effectiveness of video backtracking, the video is partially blurred to avoid user privacy leakage.

[0128] Reference below Figure 8 , which shows a schematic structural diagram of a pet identification device (eg, a computing device) 800 suitable for implementing some embodiments of the present disclosure. Figure 8 The pet identification equipment shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0129] like Figure 8 As shown, the pet identification equipment 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory 802 or a program loaded from a storage device 808 to a random access memory 803. Various programs and data required for the operation of the pet identification equipment 800 are also stored in the random access memory 803. The processing device 801, the read-only memory 802, and the random access memory 803 are connected to each other via a bus 804. An input / output interface 805 is also connected to the bus 804.

[0130] In addition, the pet identification equipment may also include: a pet locator, wherein the pet locator includes: at least two ultra-wideband signal transmitters, and the pet locator is fixed to the pet's body surface in a wearable form. A signal acquisition device, wherein the signal acquisition device includes: at least three ultra-wideband signal receivers, a video acquisition device, and an audio acquisition device, and the at least three ultra-wideband signal receivers, the video acquisition device, and the audio acquisition device face the same acquisition surface. In practice, the ultra-wideband signal transmitter is used to transmit a positioning signal. The video acquisition device and the audio acquisition device are used to collect monitoring audio and video. The ultra-wideband signal receiver is used to receive a positioning signal.

[0131] Typically, the following devices may be connected to the input / output interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the pet identification equipment 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The pet identification device 800 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0132] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a read-only memory 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0133] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0134] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0135] The computer-readable medium may be included in the pet identification device; or it may exist independently without being assembled into the pet identification device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the pet identification device, the pet identification device: in response to receiving at least one positioning signal sent by the pet identification device, determines the signal source location according to the at least one positioning signal; collects monitoring audio and video in real time according to the signal source location; determines the pet type according to the signal source location and the monitoring video included in the monitoring audio and video; in response to the pet type and the audio description information satisfying the first condition, determines the pet status description information according to the signal source location and the monitoring audio and video, wherein the audio description information is an audio description of the monitoring audio included in the monitoring audio and video, and the pet status description information includes: pet behavior description information and pet health description information; in response to the pet type and the audio description information not satisfying the first condition, determines the pet status description information according to the signal source location and the monitoring video included in the monitoring audio and video.

[0136] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor. The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0139] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A pet status recognition method, which is applied to a pet recognition device, comprising: In response to receiving at least one positioning signal sent by the pet identification device, determining a signal source location based on the at least one positioning signal; According to the signal source location, real-time monitoring audio and video is collected; Determine the type of pet based on the signal source location and the surveillance video included in the surveillance audio and video; In response to the pet type and the audio description information satisfying the first condition, pet status description information is determined according to the signal source location and the monitoring audio and video, wherein the audio description information is an audio description of the monitoring audio included in the monitoring audio and video, and the pet status description information includes: pet behavior description information and pet health description information; In response to the pet type and the audio description information not satisfying the first condition, the pet status description information is determined according to the signal source location and the surveillance video included in the surveillance audio and video.

2. The method according to claim 1, wherein: The step of determining the pet type according to the signal source location and the surveillance video included in the surveillance audio and video includes: The pet type is determined according to the signal source location, the monitoring video included in the monitoring audio and video, and the pet identity corresponding to the positioning signal, and the pet identity is bound to the identified pet target.

3. The method according to claim 1, wherein: The step of determining the signal source position according to the at least one positioning signal comprises: For each positioning signal of the at least one positioning signal, the following processing steps are performed: Determine a target information set corresponding to the positioning signal, wherein the target information in the target information set includes: a signal time difference, a first coordinate, and a second coordinate, the signal time difference represents a time difference between when a signal receiver corresponding to the first coordinate and a signal receiver corresponding to the second coordinate receive the positioning signal, Determining candidate signal source locations according to the target information set; The signal source position is corrected according to the obtained candidate signal source position set to generate the signal source position.

4. The method according to claim 1, wherein: The pet behavior description information includes: pet behavior type and pet behavior duration, wherein the pet behavior duration represents the duration of the pet maintaining the behavior corresponding to the pet behavior type, and the pet health description information includes: pet health; and The method further comprises: Accumulatively updating the pet health according to the pet behavior type and the pet behavior duration to obtain an updated pet health; In response to the updated pet health being within the first health interval, sending health abnormality prompt information to the target client; In response to the pet behavior type being an abnormal behavior type, and the pet behavior duration being greater than an abnormal behavior duration threshold corresponding to the pet behavior type, pet behavior abnormality prompt information is sent to the target client.

5. The method according to claim 1, wherein: The step of determining the pet type according to the signal source location and the surveillance video included in the surveillance audio and video includes: Extracting shallow image features from the surveillance video to obtain a shallow image feature sequence; Determine the signal source type according to the shallow image feature sequence, wherein the signal source type includes: living type and non-living type, and the living type includes: pet living type and non-pet living type; In response to the signal source type being a pet living body type, determining a region of interest according to the signal source position, wherein the region of interest includes: at least one superimposed region of interest; According to the at least one superimposed region of interest, feature windowing is performed on each shallow image feature in the shallow image feature sequence to generate a windowed shallow image feature, thereby obtaining a windowed shallow image feature sequence; Performing deep image feature extraction on the windowed shallow image feature sequence to generate deep image features, thereby obtaining a deep image feature sequence; The pet type is determined according to the deep image feature sequence.

6. The method according to claim 1, wherein: The audio description information includes: audio source description information and audio quality description information, wherein the audio source description information represents the sound source contained in the monitoring audio, and the audio description information is generated by the following steps: Performing audio feature extraction on the monitoring audio to generate audio features; Generate the audio source description information through an audio source recognition model and the audio features; The audio quality description information is generated according to the audio quality scoring model, the audio features and the audio source description information.

7. The method according to claim 1, wherein: The step of determining the pet status description information according to the signal source location and the monitored audio and video includes: Performing video feature extraction on the surveillance video to generate video features, wherein the video feature extraction model shares the shallow image feature extraction model and the deep image feature extraction model included in the pet recognition model; Performing audio feature extraction on the monitoring audio to generate audio features, wherein the audio feature extraction model is the audio feature extraction model in the audio description information generation model; Performing feature fusion on the video feature and the audio feature to generate a fusion feature; The pet status description information is generated according to the fusion feature and a pet status description information predictor, wherein the pet status description information predictor includes: a pet behavior type predictor and a pet health predictor.

8. The method according to claim 5, wherein: The method further comprises: Determine whether to allow video upload; In response to the permission, partially blurring the surveillance video to obtain a partially blurred surveillance video; Performing video compression on the locally blurred surveillance video to generate a compressed video; The compressed video is stored locally and in the cloud.

9. The method according to claim 8, wherein: The partially blurring the surveillance video to obtain the partially blurred surveillance video includes: Determine a region confidence of each of the at least one superimposed region of interest, wherein the region confidence represents a confidence that a pet is contained in the corresponding superimposed region of interest; According to the region size of the superimposed region of interest and the region confidence corresponding to the superimposed region of interest, a superimposed region of interest satisfying a second condition is screened out from the at least one superimposed region of interest as a target region of interest; According to the target region of interest, each shallow image feature in the shallow image feature sequence is inversely selected and cropped to generate cropped shallow image features, thereby obtaining a cropped shallow image feature sequence; Determine the position of the non-pet living object in the surveillance video according to the cropped shallow image feature sequence and the non-pet living object positioning model, and obtain the non-pet living position; The video is locally blurred at the non-pet living position in the surveillance video to obtain a surveillance video after the local blurring.

10. A pet identification device, comprising: A pet locator, wherein the pet locator comprises: at least two ultra-wideband signal transmitters, and the pet locator is fixed to the pet's body surface in a wearable form; A signal acquisition device, wherein the signal acquisition device comprises: at least three ultra-wideband signal receivers, a video acquisition device and an audio acquisition device, wherein the at least three ultra-wideband signal receivers, the video acquisition device and the audio acquisition device face the same acquisition surface; one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

11. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images