Pet monitoring method and related equipment

Abstract

The embodiment of the invention provides a pet monitoring method and related equipment. The pet monitoring method comprises the following steps: monitoring whether a current audio stream contains a target sound or not in real time through the sound acquisition equipment; if the current audio stream contains the target sound, controlling the image acquisition equipment to acquire a current video stream according to the target sound; according to the current video stream, monitoring model parameters are adjusted to obtain a target monitoring model; and inputting the current audio stream and the current video stream into the target monitoring model together to obtain a target monitoring result. According to the technical scheme of the embodiment of the invention, by combining the sound acquisition device and the image acquisition device, the limitation of traditional image recognition is broken through, and the behavior of the pet can still be accurately monitored under the condition that sound and image information are not completely overlapped.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a pet monitoring method and related equipment. Background Technology

[0002] Traditional pet monitoring systems typically rely solely on image capture devices to obtain information about a pet's status, neglecting the vocal signals emitted by the pet. This can sometimes prevent the system from promptly detecting unusual behavior or emotional changes in the pet, especially when the pet is making sounds but is not within the camera's field of view. For example, if a pet barks, whines, or exhibits other unusual behavior, the monitoring system may fail to respond in time without sound recognition. Summary of the Invention

[0003] The embodiments of this application provide a pet monitoring method and related equipment, which can at least to some extent overcome the above-mentioned problems of the prior art.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a pet monitoring method is provided, applied to a monitoring system including an image acquisition device, a sound acquisition device, and a server device. The pet monitoring method includes: monitoring in real time whether a target sound is contained in the current audio stream through the sound acquisition device; if the target sound is contained in the current audio stream, controlling the image acquisition device to acquire a current video stream based on the target sound; adjusting monitoring model parameters based on the current video stream to obtain a target monitoring model; and inputting the current audio stream and the current video stream together into the target monitoring model to obtain a target monitoring result.

[0006] According to one aspect of the embodiments of this application, a pet monitoring device is provided, applied in a monitoring system including an image acquisition device, a sound acquisition device, and a server device. The pet monitoring device includes: an audio acquisition module, used to monitor in real time whether a target sound is contained in the current audio stream through the sound acquisition device; a video acquisition module, used to control the image acquisition device to acquire the current video stream according to the target sound if the current audio stream contains the target sound; a parameter adjustment module, used to adjust the monitoring model parameters according to the current video stream to obtain a target monitoring model; and a model input module, used to input the current audio stream and the current video stream together into the target monitoring model to obtain a target monitoring result.

[0007] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the pet monitoring method as described in the above embodiments.

[0008] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the pet monitoring method as described in the above embodiments.

[0009] According to one aspect of the embodiments of this application, a computer program product is provided, including one or more computer programs that, when executed by one or more processors, implement the steps of the pet monitoring method as described in the above embodiments.

[0010] In some embodiments of this application, the technical solutions overcome the limitations of traditional image recognition by combining sound acquisition devices and image acquisition devices. The sound acquisition device first detects the target sound (such as a pet's bark). Upon detection of the sound signal, the image acquisition device immediately activates to acquire the relevant video stream. Next, the system adjusts the monitoring model parameters based on the video stream to ensure a high degree of matching between the image acquisition and sound recognition results, thereby improving the accuracy and real-time performance of the monitoring system. Finally, the audio and video streams are input into the target monitoring model to obtain accurate monitoring results. This approach ensures accurate monitoring of pet behavior even when sound and image information do not completely overlap.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0013] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0014] Figure 2 A flowchart illustrating a pet monitoring method provided in an embodiment of this application is shown.

[0015] Figure 3 Shown according to Figure 2 A flowchart illustrating a specific implementation of step S300 in the pet monitoring method shown in the corresponding embodiment.

[0016] Figure 4 Shown according to Figure 2 Another specific implementation flowchart of step S300 in the pet monitoring method shown in the corresponding embodiment.

[0017] Figure 5 Shown according to Figure 2 A flowchart illustrating a specific implementation of step S400 in the pet monitoring method shown in the corresponding embodiment.

[0018] Figure 6 Shown according to Figure 5 A flowchart illustrating a specific implementation of step S430 in the pet monitoring method shown in the corresponding embodiment.

[0019] Figure 7 A schematic diagram of the structure of a pet monitoring device provided in an embodiment of this application is shown.

[0020] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0022] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0023] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0024] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0025] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the object characteristics, interactive behavior characteristics, and user information involved in this specification were all obtained under full authorization.

[0026] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0027] like Figure 1 As shown, the system architecture may include audio and video acquisition devices (such as...) Figure 1 The device shown includes one or more of the following: microphone 101, mobile phone 102, and camera 103 (which could also be a desktop computer, etc.); network 104; and server 105. Network 104 serves as the medium for providing a communication link between the audio / video acquisition device and server 105. Network 104 can include various connection types, such as wired communication links, wireless communication links, etc.

[0028] It should be understood that Figure 1 The number of audio / video capture devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of audio / video capture devices, networks, and servers can be included. For example, server 105 could be a server cluster consisting of multiple servers.

[0029] Users can use audio and video capture devices to interact with server 105 via network 104 to receive or send messages, etc. Server 105 can be a server that provides various services. For example, a user can use audio and video capture device 103 (or audio and video capture device 101 or 102) to upload the current audio stream and the current video stream to server 105. Server 105 can monitor in real time whether the current audio stream contains the target sound; if the current audio stream contains the target sound, then according to the target sound, control the image capture device to acquire the current video stream; according to the current video stream, adjust the monitoring model parameters to obtain the target monitoring model; input the current audio stream and the current video stream together into the target monitoring model to obtain the target monitoring result.

[0030] It should be noted that the pet monitoring method provided in this application embodiment is generally executed by server 105, and correspondingly, the pet monitoring device is generally installed in server 105. However, in other embodiments of this application, the audio and video acquisition device may also have similar functions to the server, thereby executing the pet monitoring scheme provided in this application embodiment.

[0031] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0032] Figure 2 A flowchart of a pet monitoring method according to an embodiment of this application is shown. This pet monitoring method can be executed by a server, which may be... Figure 1 The server shown. (Refer to...) Figure 2 As shown, this pet monitoring method includes at least the following:

[0033] S100, the sound acquisition device monitors in real time whether the current audio stream contains the target sound.

[0034] S200, if the current audio stream contains the target sound, then control the image acquisition device to acquire the current video stream based on the target sound.

[0035] S300 adjusts the monitoring model parameters based on the current video stream to obtain the target monitoring model.

[0036] S400, input the current audio stream and the current video stream together into the target monitoring model to obtain the target monitoring result.

[0037] In the embodiments of this application, the limitations of traditional image recognition are overcome by combining sound acquisition devices and image acquisition devices. The sound acquisition device first detects the target sound (such as a pet's bark). Upon detection of the sound signal, the image acquisition device immediately activates to acquire the relevant video stream. Next, the system adjusts the monitoring model parameters based on the video stream to ensure a high degree of matching between the image acquisition and sound recognition results, thereby improving the accuracy and real-time performance of the monitoring system. Finally, the audio and video streams are input into the target monitoring model to obtain accurate monitoring results. This approach ensures accurate monitoring of pet behavior even when sound and image information do not completely overlap.

[0038] In the S100, the sound acquisition device continuously receives environmental audio data and uses a preset audio recognition model to analyze the audio stream in real time to detect whether it contains target sounds unique to pets (such as dog barks, cat meows, etc.). The target sound can be determined by characteristics such as sound frequency, volume, and tone.

[0039] Specifically, acoustic features such as Mel frequency cepstral coefficients (MFCC, typically 13-40 dimensions), short-time energy, and zero-crossing rate are extracted from each frame of audio. At the same time, a frequency domain spectrum is generated through Fourier transform to identify dog ​​barks and cat meows of different pitches, such as high-frequency barks of puppies and low-frequency growls of adult dogs.

[0040] Then, a pre-trained sound recognition model, such as the ResNet-18 audio version, is loaded. This model is trained on over 100,000 pet sound data points, covering the vocal features of over 20 dog breeds and over 10 cat breeds. It outputs a binary classification result (i.e., whether it is a target sound or a non-target sound) through a fully connected layer, with a threshold set to 0.9 to control the false positive rate. The recognition threshold can be automatically adjusted according to the ambient noise level (e.g., by calculating the root mean square energy of the background audio). For example, in noisy outdoor scenes, the threshold can be lowered to 0.8 to improve the tolerance for false negatives.

[0041] If the probability of detecting the target sound for more than 3 consecutive frames (i.e., 60-90ms) is greater than the threshold, it is determined that the current audio stream contains the target sound and the subsequent process is triggered; otherwise, monitoring continues.

[0042] In S200, once the target sound is confirmed, the system immediately instructs an image acquisition device (such as a surveillance camera) to start and acquire a real-time video stream related to the target sound. The image acquisition device can quickly align itself with and acquire relevant images in the current scene based on the sound location.

[0043] When a target sound is detected, the sound localization module is activated, using a microphone array time-of-arrival algorithm to calculate the azimuth and distance of the sound source. The camera's built-in stepper motor-driven pan-tilt unit calculates the angle between the current lens direction and the sound source direction based on the sound localization result. If the angle exceeds 5°, it turns towards the sound source at maximum speed. Upon reaching the target location, it triggers autofocus and automatic white balance to ensure clear image and accurate color reproduction. The image sensor begins recording at a specific frame rate, simultaneously activating the ROI (Region of Interest) function to automatically magnify the area near the sound source and reduce irrelevant background interference, ensuring the video stream contains the complete image of the sound-triggered event (such as the instant a pet begins to bark).

[0044] In the S300, video streams are fed into a pre-trained monitoring model. The system analyzes the current video content and dynamically adjusts the model's parameters based on the pet's species, behavior, and environmental changes. This helps improve the accuracy of the monitoring system, enabling it to optimize recognition based on different scenarios and pet conditions.

[0045] Specifically, in some embodiments, the specific implementation of step S300 can be found in [reference needed]. Figure 3 . Figure 3 It is based on Figure 2According to the detailed description of step S300 in the pet monitoring method shown in the corresponding embodiment, step S300 in the pet monitoring method may include the following steps:

[0046] S310, Input the current video stream into the scene classification model to obtain the current scene category.

[0047] S320: Based on the current scene category, dynamically adjust the monitoring model parameters to obtain the target monitoring model.

[0048] In this embodiment, by introducing a scene classification model, the system can determine the specific scene category after acquiring the current video stream. This determination helps the system understand environmental changes and dynamically adjust the parameters of the monitoring model according to the characteristics of different scenes, adapting it to the monitoring needs of different scenarios. In this way, the system can not only process a single video stream but also perform real-time optimization based on environmental changes, ensuring more accurate and flexible monitoring results.

[0049] Specifically, the scene classification model adjusts the monitoring model based on changes in the current scene, enabling the monitoring system to adapt to diverse scenarios. Simultaneously, by optimizing the monitoring model's parameters in dynamic scenes, the system's accuracy in recognizing pet behavior and its real-time response capability can be improved. By combining audio and image signals, the system can intelligently judge and adjust, no longer relying solely on fixed monitoring parameters, thus better adapting to complex environments.

[0050] In S310, the currently acquired real-time video stream is input into a pre-trained scene classification model. This model classifies the current scene based on the content in the video stream (such as pets, environment, objects, etc.). The classification results can include various categories such as indoor scenes, outdoor scenes, moving scenes, and static scenes.

[0051] Specifically, keyframes are extracted from the current video stream at a predetermined frequency of frames per second, reducing computational load while preserving scene features. Keyframes can be frames with significant motion changes detected using optical flow methods.

[0052] Keyframes are adjusted to a predetermined resolution, pixel values ​​are normalized to [-1, 1], and data augmentation (such as random cropping and horizontal flipping) is used to improve model robustness. MobileNetV3-Large is used as the feature extractor, and its depthwise separable convolutional structure balances accuracy and speed, making it suitable for real-time inference on edge devices.

[0053] The backbone network outputs an n×n×m feature map, which is transformed into an m-dimensional vector after global average pooling. This vector is then reduced to an a-dimensional hidden layer through a fully connected layer (with the activation function ReLU), and then mapped to the number of scene categories (e.g., 8 preset categories: indoor family, outdoor park, farm cages, nighttime environment, pet store showroom, grass, concrete, and wooden floor). The loss function is cross-entropy loss, and the training data is based on more than 100,000 labeled scene images (including different lighting, backgrounds, and pet densities).

[0054] Input each of the predetermined keyframes into the model, and take the majority vote result as the final scene category to avoid interference from single-frame noise. Taking 5 predetermined frames as an example, if 3 frames are determined to be a farm cage and 2 frames are determined to be a home interior, then the output is a farm cage.

[0055] In S320, once the scene classification model identifies the current scene category, the monitoring system dynamically adjusts the model's parameters based on this information. For example, if the system identifies a moving scene, it may need to speed up image processing to track the pet's rapid movements in real time; conversely, in a static scene, the processing frequency can be reduced to conserve computing resources. Ultimately, after scene classification and monitoring model parameter adjustments, the system forms a target monitoring model that can dynamically respond to the current environment, identify the pet's behavior, and provide corresponding feedback. This feedback may include sending alarms and recording abnormal behavior.

[0056] Specifically, in some embodiments, the specific implementation of step S320 can be found in the following embodiments. This embodiment is based on... Figure 3 According to the detailed description of step S320 in the pet monitoring method shown in the corresponding embodiment, step S320 in the pet monitoring method may include the following steps:

[0057] Based on the current scenario category, the monitoring model parameters are adjusted according to a preset parameter mapping table.

[0058] The weights of the fully connected layers of the monitoring model are updated using the predetermined frames of the current video stream.

[0059] By minimizing the detection error of known objects within the scene, a target monitoring model is obtained.

[0060] In this embodiment, by introducing scene category recognition and model parameter adjustment mechanisms, the system can dynamically adjust the parameters of the monitoring model according to the characteristics of different scenes. Specifically, the system adjusts the weights of the fully connected layers of the monitoring model according to the scene category and uses pre-defined frames in the video stream for real-time updates to optimize the detection accuracy of known objects (such as pets). This technology not only improves the accuracy of monitoring but also adapts to environmental changes, enhancing the system's flexibility and stability.

[0061] Specifically, based on historical scene data (such as over 100,000 labeled scene samples), 8-12 typical scene categories are determined through cluster analysis (K-means), with each category corresponding to a set of optimal model parameters. Then, a combination of expert knowledge and machine learning is used to define mapping rules. For sound model parameters, these may include sound source localization beamwidth, noise filtering threshold, Mel-frequency window length, etc.; for image model parameters (i.e., video frames in the video stream), these may include the target size range, non-maximum suppression (NMS) threshold, feature pyramid level, etc.

[0062] After obtaining the current scene category, the camera quickly locates the corresponding parameter group in the mapping table using a hash lookup. Specifically, for the sound model, the beam pattern of the microphone array can be adjusted according to beamforming_width, and the filter coefficients can be reconstructed in real time by a digital signal processing chip; for the image model, detection_scale can be used as a scaling factor for the YOLOv8 input size, and the IOU threshold parameters of the NMS layer can be updated. For example, the original input is 640×640, and after scaling, it becomes 768×768.

[0063] Take the first 5-10 frames of the video stream to ensure sufficient scene features and manageable latency. The specific frame count can be adjusted based on the dynamic speed of scene changes; for example, take 10 frames for outdoor scenes and 5 frames for indoor scenes.

[0064] For each frame, object detection is performed to extract the bounding box, species label, and environmental objects of the pet. Species labels include, for example, Beagle, Bernese Mountain Dog, Shaanxi Hound, Persian Cat, and British Shorthair. Environmental objects include, for example, grass, cage, and food bowl. A 1024-dimensional feature vector is extracted using the backbone network of an image model (such as the C3 module of YOLOv8) to represent the pet's appearance and environmental context.

[0065] The convolutional layer weights are fixed (pre-trained on the COCO dataset), and only the fully connected layers are open for fine-tuning to reduce computation and avoid catastrophic forgetting. These fully connected layers include either a classification head or a regression head. Mini-batch stochastic gradient descent is used with a learning rate of 0.001 and iterations limited to 3-5 times to ensure sub-second updates. For classification tasks, if common pet species are known in the scene (e.g., a breeding farm mainly uses Golden Retrievers and Shiba Inus), label-smoothed cross-entropy loss is used to force the model to output high confidence for known species. For regression tasks, mean squared error is used for sound source localization errors to reduce the deviation between predicted and actual values.

[0066] A predefined set of known objects is defined by scene categories. Known objects in the previous frames are detected using the current model parameters, and the average intersection-over-union (IoU) and classification confidence of each object are recorded. For image models, if the IoU is insufficient, the proportion of anchor frames that are close to the cage size is increased (e.g., from [16,32] to [32,64]), and the weights of lower-level features in the feature pyramid are increased to improve the detection capability of large objects. For sound models, if the sound source localization error is greater than 1 meter, a multipath propagation compensation algorithm is enabled, and the arrival time difference calculation result is corrected based on the scene reflection coefficient (e.g., the sound wave reflectivity of a metal cage is 0.9). Simultaneously, Pareto optimization is introduced to minimize the error of known objects while reducing the false detection rate of non-target objects. For example, when improving cage detection accuracy, it avoids misclassifying wooden boxes in the background as cages.

[0067] Specifically, in some other embodiments, the specific implementation of step S320 can be found in the following embodiments. This embodiment is based on... Figure 3 According to the detailed description of step S320 in the pet monitoring method shown in the corresponding embodiment, step S320 in the pet monitoring method may include the following steps:

[0068] Obtain user preference parameters and generate user preference vectors.

[0069] The current scene category is embedded into the user preference vector to obtain the scene-preference embedding vector.

[0070] Based on the scenario-preference embedding vector, the fully connected layer and output layer of the monitoring model are adjusted to obtain the target monitoring model.

[0071] In this embodiment, the monitoring model is dynamically adjusted by combining user preferences and scene categories, thereby providing a more personalized and efficient pet monitoring effect. Specifically, the system first obtains the user's preference parameters and generates a preference vector, then combines the current scene category information and embeds it into the user's preference vector. Finally, the fully connected layer and output layer of the monitoring model are adjusted to obtain the target monitoring model. This approach can not only adaptively adjust according to environmental changes, but also optimize the monitoring effect according to the user's personalized needs, improving the system's flexibility and user experience.

[0072] Specifically, user preference parameters can include customizable parameters through the app settings interface, such as attention priority, information granularity, and notification time period. Attention priority refers to behavior categories ranked by importance, such as fighting > getting lost > eating; information granularity refers to the level of detail, such as a summary of key events or a complete description including environmental details; and notification time period specifies the time period for receiving notifications, such as receiving only emergency alarms during working hours.

[0073] User preference parameters can also include the system's automatically recorded historical user interaction behavior, including notification click-through rate, video playback duration, and feedback actions. Notification click-through rate, for example, is the frequency of clicks on notifications of a specific tag type (such as injury notifications) or the frequency of clicks within a specific time period; video playback duration, for example, is the playback duration of videos of a specific tag type (such as fight videos) or the playback duration within a specific time period; feedback actions, for example, are marking false alarms or stopping notifications for a specific tag type.

[0074] Discrete preferences are converted into ordinal codes, and continuous preferences are converted into time window vectors. Historical behavioral data is used to extract keyword weights using the TF-IDF algorithm, and the mixed features are mapped into user preference vectors using a fully connected neural network.

[0075] Scene categories are encoded using one-hot encoding, converting them into binary vectors of the corresponding dimension, i.e., one-hot vectors. These one-hot vectors are then mapped to scene feature vectors using a trainable embedding matrix. The user preference vector is concatenated with the scene feature vector to form a scene-preference concatenated vector. An attention mechanism is used to calculate the weights of each dimension, highlighting key features, ultimately yielding a scene-preference embedding vector that represents the user's personalized needs in a specific scene.

[0076] Specifically, the scene-preference embedding vector can be obtained using the following formula:

[0077] S′=E·S

[0078] A=Softmax(W1·Tanh(W2·[U;S ′ ])+b)

[0079] V = A⊙[U;S ′ ]

[0080] Where S is the one-hot vector, S′ is the scene feature vector, E is the embedding matrix, [U;S′] is the scene-preference concatenation vector, A is the weight distribution after attention mechanism, ⊙ is element-wise multiplication, V is the scene-preference embedding vector, W1 is the attention weight matrix, W2 is the hidden layer weight matrix, and b is the bias vector.

[0081] The fully connected layer of the monitoring model employs a Conditional Parameter Network (CPN), whose weights are generated from the embedding vectors. Specifically, a multilayer perceptron can be used to map the embedding vectors to the target weight matrix. The loss function sets dynamic weights for different behavior categories, with the weight values ​​determined by the corresponding dimension of the embedding vector.

[0082] Based on the user's preferred information granularity parameter, the level of detail in the output text is adjusted. In simplified mode, only key tags are output; in detailed mode, a detailed report is output in conjunction with the environment description.

[0083] Specifically, in other embodiments, the specific implementation of step S300 can be found in [reference needed]. Figure 4 . Figure 4 It is based on Figure 2 According to the detailed description of step S300 in the pet monitoring method shown in the corresponding embodiment, step S300 in the pet monitoring method may include the following steps:

[0084] S340, Input the current video stream into the scene classification model to obtain the current scene category.

[0085] S350: Dynamically adjust the monitoring model parameters according to the current scene category.

[0086] S360: Obtain user preference parameters, and adjust the output layer of the monitoring model through an attention mechanism to obtain the target monitoring model.

[0087] In the embodiments of this application, the above problems are solved by dynamically adjusting the monitoring model parameters. Specifically, the system dynamically adjusts the monitoring model based on the scene classification results of the video stream, and optimizes the model's output effect by combining scene changes and user personalized preferences. Using an attention mechanism to adjust the output layer allows for focused attention on specific areas or behaviors according to user needs, thereby making the monitoring effect more personalized and accurate. This flexible adjustment method improves the adaptability of the monitoring system and can provide customized monitoring experiences according to different environments and user needs.

[0088] In the S340, a real-time video stream is first received and fed into a scene classification model. This model classifies the scene based on the content of the video stream (such as the pet's activity area, environmental background, etc.). Scene classification may include different categories such as indoor, outdoor, pet activity area, and rest area.

[0089] Keyframes are extracted from the video stream (frames with significant changes are selected through motion detection algorithms), input into the scene classification model, and features are extracted through the MobileNetV3-Large backbone network. The Softmax activation function is used to output the probability distribution of each scene category. If the highest probability is greater than the probability threshold, the corresponding category is output; otherwise, the hybrid scene mode is triggered, and parameters of multiple scenes are loaded and fused at the same time.

[0090] The specific implementation method of S340 can be referred to S310, and will not be elaborated here.

[0091] In the S350, monitoring model parameters are dynamically adjusted based on the current scene category. Different scenes may require different monitoring strategies. For example, in an indoor environment, more precise pet behavior detection may be needed, while in an outdoor scene, more attention may be paid to the dynamic tracking of pets.

[0092] Specifically, for sound models, different filter banks can be switched according to the scene. For example, a high-pass filter can be used to suppress low-frequency noise in outdoor scenes. For image models, different detection heads can be selected through a dynamic routing mechanism. For example, a home-pet association detection head can be used in a home scene, and a cage-pet association detection head can be used in a farm.

[0093] In S360, after obtaining the scene categories and making initial adjustments to the monitoring model, it's necessary to further acquire user preference parameters, such as the details of pet behavior the user is interested in and whether real-time alerts are needed. Through an attention mechanism, the system can focus on areas or behaviors of interest to the user at the output layer. For example, a user might be more concerned about whether their pet is exhibiting abnormal behavior than normal activity.

[0094] Specifically, in some other embodiments, the specific implementation of step S360 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S360 in the pet monitoring method shown in the corresponding embodiment, step S360 in the pet monitoring method may include the following steps:

[0095] Obtain user preference parameters and normalize them into a user preference vector using a multilayer perceptron.

[0096] Based on the user preference vector, preference weights are calculated using an attention mechanism.

[0097] The preference weights are injected into the intermediate features and fully connected layers of the monitoring model.

[0098] The corresponding detection heads are dynamically added to the monitoring model according to the preference weights to obtain the target monitoring model.

[0099] In this embodiment, user preference parameters are acquired and combined with an attention mechanism to optimize the output of the monitoring model, thereby achieving personalized and refined pet monitoring. Specifically, the system uses a multilayer perceptron to transform user preferences into a preference vector, then calculates preference weights based on this vector, and adjusts the feature layer and output layer of the monitoring model. This method can dynamically adjust the working mode of the monitoring system according to the specific needs of each user, achieving the target monitoring model, thereby improving monitoring effectiveness and user experience.

[0100] Specifically, user preference parameters include discrete preference parameters, continuous preference parameters, and text-based preference parameters. Discrete preference parameters include user-defined priorities for followed categories and notification frequency; continuous preference parameters include historical notification response times and video playback progress; and text-based preference parameters include user feedback messages. One-hot encoding can be used for discrete preference parameters, parameter standardization to [-1, 1] can be performed for continuous preference parameters, and corresponding text vectors are generated using Word2Vec for text-based preference parameters. The fused feature vector is input into the hidden layer, where it undergoes feature dimensionality reduction and nonlinear transformation using the ReLU activation function, and then input into the output layer. The user preference vector is output using the Tanh activation function.

[0101] The image and sound feature maps are flattened into one-dimensional vectors and then concatenated to obtain a feature vector. A preference weight vector is generated through a dot product attention mechanism, where each element represents the importance of the corresponding feature.

[0102] For the FPN feature map of the image model, preference weights are applied along the channel dimension. After the Mel-spectrum convolutional layer of the sound model, a weight normalization layer is added, applying the corresponding frequency weights from the preference weight vector along the frequency dimension. Each row of the weight matrix of the fully connected layer corresponds to a behavior category, and is scaled using preference weights.

[0103] The basic monitoring model includes a general detection head, which is pre-trained through a model zoo; when the weight of a certain category in the preference weight is greater than the preference weight threshold, the corresponding detection head is loaded.

[0104] When adjusting the detection head, pre-trained detection head parameters can be downloaded from the server. Low-rank adaptation (LoRA) technique is used to fine-tune only the last two layers of the detection head and freeze the previous layers to reduce computation.

[0105] Finally, multi-detector fusion is performed through weighted voting of the results, which can be achieved using the following formula:

[0106]

[0107] Among them, Score Final The final score is given by H, where H is the total number of detector heads, and A is the final score. h Score corresponds to the preference weights. h The score is assigned to the corresponding detection head.

[0108] For example, the main detection head outputs a score. h =0.7), the dynamically added injury detection head outputs no wound (Score = 0.7). h =0.9), then the final result is determined by A. h Decision. If the user is not concerned about the injury, then A h=0.1, you can get the score Final =0.7×0.9+0.9×0.1=0.72.

[0109] In the S400, the current audio and video streams are synchronously input into the target monitoring model. The model combines the audio and video data for comprehensive analysis to obtain the target monitoring results. For example, the system can determine whether a pet is anxious, whether it is eating, or whether it is exhibiting abnormal behavior (such as fighting, injury, etc.).

[0110] Specifically, in some embodiments, the specific implementation of step S400 can be found in [reference needed]. Figure 5 . Figure 5 It is based on Figure 2 According to the detailed description of step S400 in the pet monitoring method shown in the corresponding embodiment, step S400 in the pet monitoring method may include the following steps:

[0111] S410, extract features of the current audio stream and the current video stream through time windows of different scales to form corresponding audio temporal features and video temporal features.

[0112] S420, the audio temporal features and video temporal features are input into the causal inference fusion unit to obtain fused features and causal confidence.

[0113] S430, the fused features and causal confidence scores are input into the target analysis network to obtain the target monitoring results.

[0114] In this embodiment, by combining the current audio and video streams and fusing their features using causal inference techniques, this technology enables precise pet monitoring in more complex environments. Specifically, by extracting the temporal features of the audio and video streams through time windows of different scales and fusing them, it is possible to better identify pet behavior and improve the accuracy and reliability of monitoring. This method can address diverse monitoring needs and has higher precision and stronger adaptability.

[0115] In S410, time windowing is applied to both audio and video streams to extract their respective temporal features. Specifically, the audio and video streams are analyzed at multiple time scales to capture features of different frequency bands or durations. In this way, various information contained in the audio and video can be captured, such as changes in sound and scene, thus providing richer data for subsequent processing.

[0116] Specifically, in some embodiments, the specific implementation of step S410 can be found in the following embodiments. This embodiment is based on... Figure 5According to the detailed description of step S410 in the pet monitoring method shown in the corresponding embodiment, step S410 in the pet monitoring method may include the following steps:

[0117] The current audio stream is input into the audio timing encoder to extract the Mel spectrum timing features according to the first window, thereby obtaining the audio timing features.

[0118] The current video stream is input into the video spatiotemporal graph encoder to extract the video spatiotemporal temporal features according to the second window, thus obtaining the video temporal features.

[0119] In this embodiment, temporal features of the audio and video streams are extracted separately using time windows of different scales, and these features are processed by a specialized encoder to better capture the dynamic changes in audio and video. Audio temporal features are extracted using Mel-spectrum extraction, and video temporal features are extracted using a spatiotemporal graph encoder, enabling the system to process sound and image information simultaneously, thereby improving the accuracy and sensitivity of pet monitoring. This method effectively enhances the system's monitoring capabilities and reliably identifies pet behavior even in complex environments.

[0120] Specifically, for the audio stream, it can be segmented into a first window (e.g., 25ms frame length, 10ms frame shift), and a Hanning window can be applied to reduce spectral leakage. Then, it is transformed to the frequency domain, and a multi-dimensional spectrum is generated using a Fast Fourier Transform. The linear spectrum is converted to a Mel spectrum using multiple Mel filter banks, and the logarithm of the filtering result is taken to enhance the vocal features. Then, local temporal features are extracted through convolutional layers, and long-range dependencies are captured by a Transformer encoder. Finally, a temporal feature vector is generated through global average pooling.

[0121] For the video stream, it can be segmented according to a second window (e.g., every 5 frames), and one video frame can be extracted from each segment. Image features are extracted using ResNet50, and then keypoints (nose, ears, paws, etc.) of the pet are detected using YOLOv8. Each point is associated with spatial coordinates and visual features. Spatial edges are defined to connect points of interest (POIs) within the same frame that are less than a predetermined number of pixels apart, with a weight equal to the reciprocal of the Euclidean distance. Temporal edges are defined to connect the same POIs in adjacent frames, with a weight equal to optical flow similarity. The video frames are then processed temporally using spatial graph convolution and temporal attention layers to obtain the video temporal features.

[0122] In step S420, the audio and video temporal features extracted in step S410 are input into a causal inference fusion processor. This processor combines these two types of features to produce a fused feature. This process analyzes the causal relationship between the audio and video using a causal inference algorithm and generates a causal confidence score for each fused feature, representing the credibility of the fused feature in judging pet behavior.

[0123] Specifically, in some embodiments, the specific implementation of step S420 can be found in the following embodiments. This embodiment is based on... Figure 5 According to the detailed description of step S420 in the pet monitoring method shown in the corresponding embodiment, step S420 in the pet monitoring method may include the following steps:

[0124] A causal graph network is constructed based on the audio and video temporal features to extract causal relationships.

[0125] Based on the causal relationship, intervention calculations are performed on the causal graph network to obtain fusion features and causal confidence.

[0126] In this embodiment, by introducing a causal inference fusion processor and constructing a causal graph network based on the temporal features of audio and video, the potential causal relationship between the two can be effectively revealed. The causal graph network helps the system understand the interaction between audio and video features, thereby providing more reliable and accurate behavior recognition. Optimizing the causal graph network through intervention computation can further improve the accuracy of pet behavior judgment, and the credibility of the prediction results can be quantified through causal confidence metrics, thereby enhancing the robustness and intelligence of the entire monitoring system.

[0127] Specifically, a causal graph network is constructed by using Mel-spectral temporal features as audio feature nodes and spatiotemporal temporal features as video feature nodes. The causal graph network includes both undirected and directed edges. Undirected edges are initialized with the cross-correlation coefficients of audio and video features to measure statistical correlation. Directed edges are determined to have causal directions using the Granger causality test. During the construction of the causal graph, neighbor features can be aggregated using a graph convolutional neural network to extract direct causal relationships. These relationships are then weighted using an attention mechanism to output a causal probability matrix, yielding the causal probabilities.

[0128] Then through do(a) i =1) indicates that the audio feature a is forcibly set. i If true, observe the video feature v. j The change can be specifically expressed using the following formula:

[0129]

[0130] Among them, v j Let a be the j-th variable in the video temporal features (e.g., the amplitude of pet movements, ambient brightness, etc.); i Let i be the i-th variable in the audio temporal features (e.g., pet barking frequency, ambient noise decibels, etc.); do(a i ) is for audio variable a i Conduct causal intervention (i.e., forcefully set a) i The value of a (excluding other confounding factors); ai ′ for a i The parent node variable (which directly affects a in the cause-effect graph) i Variables, such as the influence of pet breed on vocalization; pa(v j ) for v j The set of parent nodes (directly affecting v in the causal graph) j Variables, such as a i (or other video variables); P(v) j ||do(a i )) is given a i ′ and v j Given the parent node, v j The probability distribution of P(a) i ′) is the parent node a i The prior probability distribution of ′.

[0131] Then, the average causal effect is calculated to measure audio feature a. i For video features v j The intensity of the influence can be specifically expressed using the following formula:

[0132] ACE = E[V j ||do(A i =1)]-E[V j |do(A i =0)

[0133] Where ACE represents the average causal effect; A i For audio variable a i The binary representation of (e.g., A) i =1 indicates an abnormal sound, A i =0 indicates a normal call); V j For video variable v j The random variable form; E[·] is the mathematical expectation, representing the average result after intervention.

[0134] Then for each video feature v j Its causal weight is the normalized result of the ACE values ​​of all parent nodes, which can be expressed using the following formula:

[0135]

[0136] Among them, w j For video variable v j The fusion weight reflects its importance in the causal relationship; pa(v j ) for v j The set of audio parent nodes (i.e., for v) jAudio variables with direct causal influence); n is the total number of audio variables (e.g., sound characteristics at different frequencies); m is the total number of video variables (e.g., visual characteristics at different spatial locations); |ACE(a k ,v l ) | is the audio variable a k For video variable v l The absolute value of the causal effect is used for normalization.

[0137] Next, feature-weighted fusion is performed, which can be done using the following formula:

[0138]

[0139] Among them, F fusion The final fused feature vector integrates audio-video causal relationships; w j For video variable v j The fusion weight; For cross-modal fusion operations (such as feature concatenation, element-wise addition, or attention mechanism fusion); do(a i =1)(v j ) is for audio variable a i When the intervention is 1, the video variable v j The response characteristics (such as the pet's tense behavior corresponding to abnormal cries).

[0140] Finally, the causal confidence score is calculated, and the confidence score is normalized after obtaining the causal consistency score. The specific formula is as follows:

[0141] Consistency = NCCP / TCP

[0142] Confidence=σ(λ1·ACEmean+λ2·Consistency)

[0143] Where Confidence is the causal confidence level, Consistency is the causal path consistency, NCCP is the number of causal paths in the causal graph that conform to prior knowledge or data distribution (such as the path where the bark points to the pet's movement); TCP is the total number of all possible causal paths in the causal graph (such as the direct or indirect influence path between audio and video variables); σ is the activation function (such as Sigmoid, which maps the output to the [0,1] interval); λ1 and λ2 are hyperparameters that adjust the weights of average causal effect and consistency; ACEmean is the average causal effect value of all audio-video variable pairs, reflecting the overall causal strength.

[0144] In S430, the features and confidence scores fused through causal reasoning are input into the target analysis network. This network performs further analysis based on the input feature information and outputs the final target monitoring results. This analysis network can make accurate judgments based on the fused features, such as determining whether a pet is exhibiting abnormal behavior or whether an alarm needs to be issued.

[0145] Specifically, in some embodiments, the specific implementation of step S430 can be found in [reference needed]. Figure 6 . Figure 6 It is based on Figure 5 According to the detailed description of step S430 in the pet monitoring method shown in the corresponding embodiment, step S430 in the pet monitoring method may include the following steps:

[0146] S432, The fused features and causal confidence are input into the time series risk prediction network to obtain the time series risk.

[0147] S434, The time-series risk and fusion features are input into the multi-layer analysis network to obtain the target monitoring results, which include behavior recognition results, risk level results and environmental anomaly results.

[0148] In this embodiment, by inputting fused features and causal confidence scores into the target analysis network, combined with a temporal risk prediction network and a multi-layer analysis network, the risks of pet behavior can be analyzed more comprehensively, dynamic changes in pet behavior can be identified, and potential environmental anomalies can be determined in a timely manner. Specifically, the temporal risk prediction network predicts risks during changes in pet behavior, improving the accuracy of risk identification. Utilizing the multi-layer analysis network to process fused features allows for a comprehensive evaluation of pet behavior identification, risk level, and environmental anomalies, thereby providing comprehensive target monitoring results. This embodiment not only identifies pet behavior at multiple levels but also comprehensively predicts potential risks and makes intelligent judgments, thus optimizing the effectiveness of pet monitoring.

[0149] In S432, fused features and causal confidence are calculated by fusing audio and video features and combining them with causal inference. These fused features reflect the interaction between audio and video signals, as well as the credibility of the behavior. Inputting these features into a temporal risk prediction network allows the network to learn patterns and changes in the temporal data and predict potential risks at a future point in time. For example, in the event of a sudden change in pet behavior, the system can predict whether this change will lead to potential harm or danger.

[0150] Specifically, the fused features and causal confidence scores are concatenated, sliced ​​according to time windows, and used to form an input tensor of a predetermined sequence length. A bidirectional LSTM layer is then used to capture the temporal dependencies between the past and future, followed by an attention layer to calculate the risk weights for each time step. In the output layer, a fully connected layer and sigmoid activation are used to output the temporal risk sequence.

[0151] In S434, the temporal risk obtained from the temporal risk prediction network, along with the fused features, is input into a multi-layer analysis network. This multi-layer analysis network comprises multiple layers of processing units, each responsible for analyzing and making decisions based on different data features. First, the system performs preliminary classification of the input data to identify the type of pet behavior. Next, based on temporal risk and environmental factors, the analysis system assesses the risk level of the pet behavior. For example, one behavior may be associated with high risk, while another behavior is assessed as low risk. Finally, the system also determines whether there are any abnormal factors in the environment, such as overheating electrical appliances or abnormal temperatures at the pet's location.

[0152] The behavioral recognition results accurately identify the type of activity the pet is currently engaged in, such as eating, sleeping, playing, or seeking help. The risk level results assess whether the pet's current behavior poses a high risk and whether there is a possibility of harm, based on time-series risk predictions and environmental factors. The environmental anomaly results check for potential anomalies in the environment, such as excessively high temperatures, excessively low humidity, or malfunctioning electrical equipment, which could affect the pet's safety.

[0153] Specifically, in some embodiments, the specific implementation of step S434 can be found in the following embodiments. This embodiment is based on... Figure 5 According to the detailed description of step S434 in the pet monitoring method shown in the corresponding embodiment, step S434 in the pet monitoring method may include the following steps:

[0154] The temporal risks and fused features are input into a multi-layer cross-modal Transformer encoder to extract key temporal features.

[0155] Self-attention processing is applied to the key temporal features to obtain enhanced temporal features.

[0156] By inputting enhanced temporal features into a multi-layer analysis network, target monitoring results are obtained.

[0157] In this embodiment, by introducing a cross-modal Transformer encoder and a self-attention mechanism, key features can be effectively extracted and enhanced from time-series data, thereby improving the monitoring effect and prediction accuracy of pet behavior.

[0158] Cross-modal Transformer encoders can process different types of data (such as video and audio features) and extract the correlations between them. This fusion helps the system comprehensively understand the temporal dynamics of pet behavior. Self-attention processing of temporal features in multi-level analysis helps capture key changes in the temporal data, improving the system's sensitivity to abnormal behavior. By comprehensively processing temporal features through a multi-level analysis network, the system can provide more accurate behavior recognition, risk assessment, and environmental anomaly detection results, further enhancing the intelligence level of pet monitoring.

[0159] This embodiment extracts enhanced features from time-series data, enabling the system to perform more accurate behavior recognition, risk prediction, and environmental monitoring, thereby improving monitoring effectiveness and response speed.

[0160] In the embodiments of this application, the limitations of traditional image recognition are overcome by combining sound acquisition devices and image acquisition devices. The sound acquisition device first detects the target sound (such as a pet's bark). Upon detection of the sound signal, the image acquisition device immediately activates to acquire the relevant video stream. Next, the system adjusts the monitoring model parameters based on the video stream to ensure a high degree of matching between the image acquisition and sound recognition results, thereby improving the accuracy and real-time performance of the monitoring system. Finally, the audio and video streams are input into the target monitoring model to obtain accurate monitoring results. This approach ensures accurate monitoring of pet behavior even when sound and image information do not completely overlap.

[0161] It should be noted that, specifically, the training method for the aforementioned target monitoring model includes:

[0162] Obtain the audio and video stream sample set, which contains multiple audio and video stream samples, each of which is labeled with a corresponding target monitoring tag.

[0163] The audio and video stream samples are input one by one into the target monitoring model to obtain the target monitoring results.

[0164] Based on the output target monitoring results and the target monitoring labels, the parameters of the target monitoring model are updated until the predetermined termination condition is met, thus ending the training and obtaining the trained target monitoring model.

[0165] In the embodiments of this application, during training, an audio and video stream sample set containing multiple audio and video stream samples can be obtained first. Each audio and video stream sample contains an audio stream and a video stream, and is marked with a corresponding target monitoring label. Then, the multiple audio and video stream samples are divided into a training set, a validation set, and a test set according to a predetermined ratio. Then, the parameters of the encoder and decoder in the target monitoring model are adjusted and determined according to the audio and video stream samples included in the training set, validation set, and test set, so as to obtain the trained target monitoring model.

[0166] It should be noted that the audio and video stream samples mentioned above include video stream samples with obstructions to improve the model's ability to identify various obstructions. They also include audio stream samples with noise and interference to improve the model's ability to identify various noises and interference.

[0167] When training the model, the audio and video stream sample set can be divided into a training set, a validation set, and a test set. Then, the model is trained based on the training set, validated based on the validation set, and tested based on the test set to obtain the trained target monitoring model.

[0168] Before training on the training set, the audio and video stream image samples in the training set can be preprocessed. Preprocessing includes image and sound adjustments, normalization, data augmentation, and class encoding.

[0169] The specific methods for data augmentation include performing geometric transformations (including flipping, rotating, deforming, scaling, etc.) on the data image, as well as color transformations (including noise reduction, blurring, color transformation, erasing, and filling). These operations can better expand the shape of the video frame image.

[0170] In some embodiments, GAN networks can also be used to generate images of target pets that are occluded, out of focus, or blurred, making them difficult to identify but hard to obtain. This helps the algorithm model accurately identify the target pet, reduces false positives and false negatives, and improves the model's accuracy in identifying target pets from special images. GAN networks can also be used to generate audio from noisy environments or with multiple sound sources, making them difficult to identify but hard to obtain. This helps the algorithm model accurately identify the target pet, reduces false positives and false negatives, and improves the model's accuracy in identifying target pets from special audio.

[0171] After obtaining the enhanced training set, the target monitoring model can be trained based on the enhanced training set, and the parameters and weights in the network can be updated.

[0172] Specifically, audio and video stream image samples from the training set are input into the target monitoring model to obtain the target monitoring results output by the model. The target monitoring results are compared with the target monitoring labels, and a loss function is calculated. Then, the stochastic gradient descent method is used to minimize the loss function. The parameters and weights in the target monitoring model are updated via backpropagation until the loss function meets predetermined conditions, such as convergence or falling below a predetermined threshold.

[0173] In some embodiments, the target monitoring results include behavior recognition results, risk level results, and environmental anomaly results. The target monitoring label may include at least one of the behavior recognition label, risk level label, and environmental anomaly label. The loss function is calculated by comparing the risk level results with the risk level label and / or comparing the behavior recognition results with the behavior recognition label and / or comparing the environmental anomaly results with the environmental anomaly label.

[0174] After training, the target monitoring network with updated parameters from the training set can be validated using a validation set. Specifically, the target monitoring network is debugged based on the validation set data. When the loss function meets predetermined conditions, the model parameters for that stage are output. If the loss function does not meet the predetermined conditions, hyperparameters such as the learning rate are automatically adjusted, and the next round of network model training is performed.

[0175] When the loss function computed on the validation set meets predetermined conditions, the parameters and weights can be retained, and then the retained parameters are tested based on the test set. Specifically, the input test set data enables the target monitoring network with retained parameters and weights to output target monitoring results and model weights. By comparing the model loss and corresponding weights across multiple rounds, the model weights with the minimum loss are output, thus determining the trained target monitoring model.

[0176] Once the trained target monitoring model is obtained, target monitoring can be performed based on the target monitoring model.

[0177] Furthermore, data augmentation is not required for the input data in the validation and test sets.

[0178] The following describes an embodiment of the apparatus described in this application, which can be used to execute the pet monitoring method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the pet monitoring method described above.

[0179] Figure 7 A block diagram of a pet monitoring device according to an embodiment of this application is shown.

[0180] Reference Figure 7As shown, a pet monitoring device 700 according to an embodiment of this application is applied to a monitoring system including an image acquisition device, a sound acquisition device, and a server device. The pet monitoring device includes: an audio acquisition module 710, a video acquisition module 720, a parameter adjustment module 730, and a model input module 740.

[0181] The audio acquisition module 710 is used to monitor in real time whether the current audio stream contains the target sound through the sound acquisition device; the video acquisition module 720 is used to control the image acquisition device to acquire the current video stream according to the target sound if the current audio stream contains the target sound; the parameter adjustment module 730 is used to adjust the monitoring model parameters according to the current video stream to obtain the target monitoring model; and the model input module 740 is used to input the current audio stream and the current video stream together into the target monitoring model to obtain the target monitoring result.

[0182] In some feasible embodiments of this application, the parameter adjustment module 730 specifically includes: a scene classification submodule, used to input the current video stream into a scene classification model to obtain the current scene category; and a dynamic adjustment submodule, used to dynamically adjust the monitoring model parameters according to the current scene category to obtain the target monitoring model.

[0183] In some feasible embodiments of this application, the dynamic adjustment submodule specifically includes: a parameter adjustment unit, used to adjust the monitoring model parameters according to the current scene category and a preset parameter mapping table; a weight update unit, used to update the fully connected layer weights of the monitoring model using the previous predetermined frames of the current video stream; and a detection error unit, used to minimize the detection error of known objects in the scene to obtain the target monitoring model.

[0184] In some feasible embodiments of this application, the dynamic adjustment submodule specifically includes: a preference acquisition unit, used to acquire user preference parameters and generate a user preference vector; a vector embedding unit, used to embed the current scene category into the user preference vector to obtain a scene-preference embedding vector; and a weight adjustment unit, used to adjust the fully connected layer and output layer of the monitoring model according to the scene-preference embedding vector to obtain a target monitoring model.

[0185] In some feasible embodiments of this application, the parameter adjustment module 730 specifically includes: a scene classification submodule, used to input the current video stream into a scene classification model to obtain the current scene category;

[0186] The parameter adjustment submodule is used to dynamically adjust the parameters of the monitoring model according to the current scene category; the weight adjustment submodule is used to obtain user preference parameters and adjust the output layer of the monitoring model through an attention mechanism to obtain the target monitoring model.

[0187] In some feasible embodiments of this application, the weight adjustment submodule specifically includes: a preference vector unit, used to obtain user preference parameters and normalize the user preference parameters into a user preference vector through a multilayer perceptron; a preference weight unit, used to calculate preference weights based on the user preference vectors through an attention mechanism; a weight injection unit, used to inject the preference weights into the intermediate layer features and fully connected layers of the monitoring model; and an output update unit, used to dynamically add corresponding detection heads to the monitoring model according to the preference weights to obtain the target monitoring model.

[0188] In some feasible embodiments of this application, the model input module 740 specifically includes: a temporal feature submodule, used to extract features of the current audio stream and the current video stream through time windows of different scales, forming corresponding audio temporal features and video temporal features; a causal inference submodule, used to input the audio temporal features and video temporal features into a causal inference fusion processor to obtain fused features and causal confidence; and a target analysis submodule, used to input the fused features and causal confidence into a network to obtain target monitoring results.

[0189] In some feasible embodiments of this application, the timing feature submodule specifically includes: an audio timing unit, used to input the current audio stream into an audio timing encoder to extract Mel spectrum timing features according to a first window, thereby obtaining audio timing features; and a video timing unit, used to input the current video stream into a video spatiotemporal graph encoder to extract video spatiotemporal timing features according to a second window, thereby obtaining video timing features.

[0190] In some feasible embodiments of this application, the audio temporal features and video temporal features are input into a causal inference fusion processor to obtain fused features and causal confidence scores. Specifically, this includes: a causal relationship unit, used to construct a causal graph network based on the audio temporal features and video temporal features, and extract causal relationships; and an intervention calculation unit, used to perform intervention calculations on the causal graph network based on the causal relationships, and obtain fused features and causal confidence scores.

[0191] In some feasible embodiments of this application, the step of inputting the fusion features and causal confidence scores into the target analysis network to obtain target monitoring results specifically includes: a risk prediction unit, used to input the fusion features and causal confidence scores into a time-series risk prediction network to obtain time-series risk; and a multi-layer analysis unit, used to input the time-series risk and fusion features into a multi-layer analysis network to obtain target monitoring results, wherein the target monitoring results include behavior recognition results, risk level results, and environmental anomaly results.

[0192] In some feasible embodiments of this application, the multi-layer analysis unit specifically includes: a feature extraction subunit, used to input the temporal risk and fused features into a multi-layer cross-modal Transformer encoder to extract key temporal features; a feature enhancement subunit, used to perform self-attention processing on the key temporal features to obtain enhanced temporal features; and a multi-layer analysis subunit, used to input the enhanced temporal features into a multi-layer analysis network to obtain target monitoring results.

[0193] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0194] It should be noted that Figure 8 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0195] like Figure 8 As shown, the computer system includes a Central Processing Unit (CPU) 1801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1802 or programs loaded from storage portion 1808 into Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. The RAM 1803 also stores various programs and data required for system operation. The CPU 1801, ROM 1802, and RAM 1803 are interconnected via a bus 1804. An Input / Output (I / O) interface 1805 is also connected to the bus 1804.

[0196] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.

[0197] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by central processing unit (CPU) 1801, it performs various functions defined in the system of this application.

[0198] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a camera disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0200] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0201] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0202] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-6 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-6 The specific details of the illustrated embodiments will not be elaborated here.

[0203] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0204] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0205] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0206] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A pet monitoring method, characterized in that, The pet monitoring method, applied in a monitoring system that includes image acquisition devices, sound acquisition devices, and server-side devices, comprises: The sound acquisition device monitors in real time whether the current audio stream contains the target sound. If the current audio stream contains the target sound, then the image acquisition device is controlled to acquire the current video stream based on the target sound; Based on the current video stream, adjust the monitoring model parameters to obtain the target monitoring model; The current audio stream and the current video stream are input together into the target monitoring model to obtain the target monitoring result.

2. The pet monitoring method as described in claim 1, characterized in that, The step of adjusting the monitoring model parameters based on the current video stream to obtain the target monitoring model specifically includes: Input the current video stream into the scene classification model to obtain the current scene category; Based on the current scenario category, the monitoring model parameters are dynamically adjusted to obtain the target monitoring model.

3. The pet monitoring method as described in claim 2, characterized in that, The step of dynamically adjusting the monitoring model parameters according to the current scene category to obtain the target monitoring model specifically includes: Based on the current scenario category, adjust the monitoring model parameters according to the preset parameter mapping table; Update the weights of the fully connected layers of the monitoring model using the previous frames of the current video stream; By minimizing the detection error of known objects within the scene, a target monitoring model is obtained.

4. The pet monitoring method as described in claim 2, characterized in that, The step of dynamically adjusting the monitoring model parameters according to the current scene category to obtain the target monitoring model specifically includes: Obtain user preference parameters and generate user preference vectors; Embed the current scene category into the user preference vector to obtain the scene-preference embedding vector; Based on the scenario-preference embedding vector, the fully connected layer and output layer of the monitoring model are adjusted to obtain the target monitoring model.

5. The pet monitoring method as described in claim 1, characterized in that, The step of adjusting the monitoring model parameters based on the current video stream to obtain the target monitoring model specifically includes: Input the current video stream into the scene classification model to obtain the current scene category; The monitoring model parameters are dynamically adjusted based on the current scenario category. By acquiring user preference parameters and adjusting the output layer of the monitoring model through an attention mechanism, the target monitoring model is obtained.

6. The pet monitoring method as described in claim 5, characterized in that, The process of obtaining user preference parameters and adjusting the output layer of the monitoring model through an attention mechanism to obtain the target monitoring model specifically includes: Obtain user preference parameters and normalize them into a user preference vector using a multilayer perceptron. Based on the user preference vector, preference weights are calculated using an attention mechanism; The preference weights are injected into the intermediate features and fully connected layers of the monitoring model; The corresponding detection heads are dynamically added to the monitoring model according to the preference weights to obtain the target monitoring model.

7. A pet monitoring device, characterized in that, The pet monitoring device, applied in a monitoring system that includes image acquisition equipment, sound acquisition equipment, and server equipment, comprises: The audio acquisition module is used to monitor in real time whether the current audio stream contains the target sound through the sound acquisition device; The video acquisition module is used to control the image acquisition device to acquire the current video stream based on the target sound if the current audio stream contains the target sound; The parameter adjustment module is used to adjust the monitoring model parameters according to the current video stream to obtain the target monitoring model; The model input module is used to input the current audio stream and the current video stream into the target monitoring model to obtain the target monitoring result.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the pet monitoring method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: one or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the pet monitoring method as described in any one of claims 1 to 6.

10. A computer program product comprising one or more computer programs, characterized in that, When the one or more computer programs are executed by one or more processors, they implement the steps of the pet monitoring method according to any one of claims 1 to 6.

Images