Behavior analysis and correction system based on pet visual angle shooting data

Abstract

The invention belongs to the technical field of information, and discloses a behavior analysis and correction system based on pet visual angle shooting data. Comprising an acquisition processing module which acquires a pet visual angle video, acceleration data, gyroscope data and behavior response time, and obtains video frame data, acceleration features and gyroscope features through preprocessing; the first analysis module is used for taking the video frame data as the input of a target detection model to obtain the probability containing the pet partner face and bounding box coordinates; based on a pet visual angle video, acceleration data and gyroscope data are combined, and behaviors of a pet can be observed and analyzed from the visual angle of the pet in an all-around mode; and pet individual labels of different pet individuals can be further analyzed based on the collected pet view angle video, and a behavior correction strategy fitting the individual is formulated according to the pet individual labels.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and more specifically, to a behavior analysis and correction system based on data shot from a pet's perspective. Background Art

[0002] For most pet-raising families, there will be situations where the pets stay alone at home. At this time, the owners cannot know the status of the pets, which inevitably makes them worried. Therefore, most pet owners install surveillance cameras at home to remotely monitor the status of their pets. The accuracy and diversity of behavior recognition are insufficient. Existing behavior analysis algorithms mainly rely on static feature extraction methods, which make it difficult to accurately identify the complex dynamic behaviors of pets. For example, subtle emotional changes (such as tension and excitement) and different behavior patterns (such as aggression or play) are often misclassified. The correction method is single, and there is a lack of personalized correction solutions that combine the individual characteristics of pets.

[0003] The Chinese patent application with publication number CN118097779A discloses a pet behavior recognition method and electronic device, including: inputting pet data into a pre-trained pet behavior recognition model; extracting pet data using a feature processing module to obtain pet features; classifying the pet features using a species classification module to obtain pet categories; based on the pet category, using a facial detection and emotion recognition module to perform emotion recognition on the pet features to obtain emotion categories; using a posture estimation module to perform posture estimation on the pet features to obtain coordinate information of each feature point; based on the pet category, emotion category, and coordinate information of each feature point, using a comprehensive behavior analysis module to determine the behavior category, and using the behavior category as the pet behavior recognition result. The invention combines the advantages of convolution and Transform structures, can take into account both the overall and local features of the image, improve the analysis accuracy of animal behavior recognition based on the image, and reduce the amount of network inference calculation. In addition, by adding facial emotion recognition of the pet and combining it with the emotional characteristics of the pet, the pet behavior can be better analyzed. And there is no need to wear sensors, no need to purchase additional accessories, and will not affect the living habits of the pet itself.

[0004] Although the above method can meet most scenarios, research and practical application of the above method and existing technology have found that the above method and existing technology have at least the following defects:

[0005] 1. It mainly relies on conventional video cameras, which usually shoot pets from an external, relatively fixed angle. The data obtained is relatively single-angle and cannot accurately reflect the content that pets pay attention to and the behavioral triggers. It is easy to miss the details that are of great value to the pet behavior analysis, making the subsequent interpretation of the pet's behavior not comprehensive and accurate.

[0006] 2. Based on preset, relatively fixed environmental settings and behavior judgment standards, the system lacks sensitivity and adaptability to environmental changes. Once the actual environment of the pet is significantly different from the system preset environment, the system may not be able to accurately identify the pet's behavior and judge the nature of the behavior, making the behavior correction function ineffective and reducing the practicality and reliability of the system in actual applications.

[0007] 3. Failure to take into account the differences between individual pets will result in unsatisfactory correction effects on different pets, which may even trigger resistance in the pets and further worsen behavioral problems. This is not conducive to establishing a good pet behavior management model and it is difficult to achieve accurate and effective behavior correction.

[0008] In view of this, the present invention proposes a behavior analysis and correction system based on pet-perspective-shot data to solve the above-mentioned problems. Summary of the invention

[0009] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a behavior analysis and correction system based on pet-perspective-shot data, comprising:

[0010] Acquisition and processing module: collects pet-perspective video, acceleration data, gyroscope data and behavioral reaction time, and obtains video frame data, acceleration features and gyroscope features through preprocessing;

[0011] The first analysis module: using the video frame data as the input of the target detection model to obtain the probability of containing the pet playmate's face and the bounding box coordinates; marking the video frame data with a probability higher than a preset probability threshold as the first data, marking the remaining video frame data as the second data, extracting the key point features of the pet playmate's face in the first data; using the key point features of the face as the input of the emotion recognition model to obtain the emotion recognition result of the pet playmate;

[0012] The second analysis module: the second data is used as the input of the video analysis model to obtain the video analysis features; the video analysis features are used as the input of the time series model to obtain the video features; the video analysis model is a multi-layer CNN network, and the CNN network includes a de-jitter layer and NH convolutional layers; the de-jitter layer is composed of a dual-stream processing structure consisting of an optical flow estimation sub-network and a sensor fusion sub-module; the convolution kernel size of each convolution layer in the NH convolutional layers decreases as the number of convolutional layers increases, and each convolutional layer is connected to a hole convolution module in which the convolution kernel is integrated into a deformable mechanism;

[0013] Behavior analysis module: The behavior analysis features are used as inputs of the behavior recognition model and the risk analysis model to obtain the corresponding pet behavior recognition results and risk assessment scores; the behavior analysis features include the emotion recognition results, video features, acceleration features, and gyroscope features of the pet playmate;

[0014] The third analysis module: taking the behavior reaction time as the input of the personality analysis model, obtaining the pet personality label i, and defining the correction strategy library B for the pet personality label i;

[0015] Behavior correction module: The pet personality label i, behavior recognition results and risk assessment scores are used as inputs to the reinforcement learning algorithm to obtain the behavior correction strategy.

[0016] Furthermore, the video analysis model is a multi-level CNN network model, which includes a de-jitter layer, an input layer, NH convolutional layers, a pooling layer, a fully connected layer and an output layer; the de-jitter layer is composed of a dual-stream processing structure consisting of an optical flow estimation subnetwork and a sensor fusion submodule, wherein the optical flow estimation subnetwork adopts a lightweight FlowNet structure to calculate the first motion compensation amount through the optical flow field between adjacent frames: the sensor fusion submodule inputs acceleration data and gyroscope data into the LSTM network to predict the device motion trajectory to obtain the second motion compensation amount; the de-jitter layer adopts differentiable bilinear sampling combined with the first motion compensation amount and the second motion compensation amount to calculate the total compensation amount for motion compensation; in the NH convolutional layers, the convolution of each convolutional layer The convolution kernel size decreases with the increase of the number of convolution layers, among which the convolution kernel size of the first convolution layer is the largest, and the convolution kernel size of the NHth convolution layer is the smallest. A dilated convolution module is connected to each convolution layer; the output of the dilated convolution module connected to the nhth convolution layer is obtained by convolution operation on the corresponding input, nh=1,…,NH; the convolution operation is used to perform weighted summation on each element position of the corresponding input; during the convolution operation, the interval between elements is controlled by the dilation rate; the convolution kernel of the dilated convolution module is integrated into the deformable mechanism, and an offset prediction branch is added to the convolution kernel of each dilated convolution module. By obtaining the offset of each sampling position of the convolution kernel, the offset is superimposed on the size of the convolution kernel of the original dilated convolution module to obtain the deformed convolution kernel of the dilated convolution module.

[0017] Furthermore, the correction actions of the correction strategy library B include: actions in the sound prompt correction method, actions in the device vibration correction method and actions in the owner intelligent notification method; in the sound prompt correction method, the sound prompt correction action is synthesized by different sound frequencies and sound volumes; in the device vibration correction method, the device vibration correction action is synthesized by different vibration intensities and vibration frequencies.

[0018] Further, the method for obtaining the behavior correction strategy includes:

[0019] The behavior correction module is defined as an agent, the pet and its environment are defined as the environment, the pet's pet personality tag, behavior recognition result and risk assessment score are defined as the environment state, and the correction action corresponding to the pet personality tag in the correction strategy library B is defined as an action; and a reward function for evaluating the value of an action is also defined;

[0020] Use the Q learning algorithm to learn the optimal strategy, initialize the Q value function as a table, the rows of the table correspond to different environmental states s, the columns correspond to different actions e, set the learning rate λ, discount factor δ and exploration rate ∈; at each time step t, observe the pet’s current environmental state s t , according to the exploration rate ∈, randomly select an action with probability ∈, and select the action with the largest Q value in the current environment state with probability 1-∈; execute the selected action, observe the pet's reaction, obtain new behavior recognition results and new risk assessment scores through the behavior analysis module, and obtain the new environment state s t+1 ; At the same time, the immediate reward R is calculated according to the reward function t+1 ;

[0021] Repeat the Q value function until the Q value converges to obtain the corresponding optimal Q value function. When the behavior analysis module detects that the pet's behavior needs to be corrected, the optimal Q value function is used to select the optimal action as the correction strategy from the correction strategy library corresponding to the pet's personality label i in the environmental state according to the environmental state and execute it;

[0022] The update of the Q value function is specifically as follows:

[0023] The updated Q value is the sum of the current Q value and the correction term. The correction term is the product of the learning rate and the difference. The difference is the immediate reward plus the discounted maximum Q value of the next state, minus the current Q value.

[0024] Furthermore, the reward function is a weighted combination of behavior frequency and stop frequency, wherein the behavior frequency is the ratio of the difference in the number of times the pet performs a specific behavior before and after the implementation of the behavior correction strategy to the number of times the pet performs the specific behavior before the implementation of the behavior correction strategy; the stop frequency is the ratio of the difference in the number of times the pet spontaneously stops a specific behavior before and after the implementation of the behavior correction strategy to the number of times the pet spontaneously stops the specific behavior before the implementation of the behavior correction strategy.

[0025] Furthermore, the behavior reaction time of the pet is the behavior reaction time of the pet in G types of stimulation scenes, and the method for collecting the behavior reaction time of the pet in G types of stimulation scenes includes:

[0026] Set the types and parameters of G stimulation scenes. For the Vth type of stimulation scene among the G stimulation scenes, accurately record the time t1 when the stimulation starts, and also record the time t2 when the pet shows a behavioral response. Repeat the recording n times. t times, n t The average value of the difference between t1 and t2 is taken as the behavioral reaction time of the pet in the Vth type of stimulation scenario; wherein, when the acceleration data of the pet is higher than the preset acceleration threshold or the gyroscope data is higher than the preset gyroscope threshold, it indicates that the pet has a behavioral reaction.

[0027] Further, the pet perspective video includes video data and its corresponding time data;

[0028] Methods for obtaining video frame data include:

[0029] The acquired video data is obtained in the order of time data to obtain the corresponding video frame sequence. For each frame of video data, all its pixels are traversed, and the square of the difference between the corresponding pixels of the t-th frame and the t+1-th frame is calculated. The square of the difference of all pixel points is summed, and the sum is divided by the product of the width, height and number of channels of the video data to obtain the pixel difference between the t-th frame and the t+1-th frame of video data;

[0030] The pixel difference between the t-th frame and the t+1-th frame video data is compared with a preset pixel difference threshold, and when the pixel difference is not less than the preset pixel difference threshold, the t+1-th frame is extracted to obtain video frame data.

[0031] Furthermore, the method of obtaining the acceleration characteristic and the gyroscope characteristic includes:

[0032] The time corresponding to each frame of video frame data in the video frame data is counted, the average acceleration data and the average gyroscope data corresponding to each frame of video frame data are calculated, and the calculated average acceleration data and average gyroscope data sequences are used as acceleration features and gyroscope features.

[0033] Furthermore, in the behavior analysis module, whether the pet behavior needs to be corrected is determined based on the comparison result between the risk assessment score and the preset risk threshold. When the risk assessment score exceeds the preset risk threshold, it is determined that the pet behavior needs to be corrected, otherwise the pet behavior does not need to be corrected.

[0034] Furthermore, the facial key point features include eye features, mouth features, and ear features. The eye features include the coordinates of the inner corner of the left eye, the outer corner of the left eye, the center of the pupil of the left eye, the inner corner of the right eye, the outer corner of the right eye, and the center of the pupil of the right eye; the mouth features include the coordinates of the left corner of the mouth, the midpoint of the upper lip, the midpoint of the lower lip, and the coordinates of the right corner of the mouth; the ear features include the coordinates of the left ear tip, the right ear tip, the left ear base, and the right ear base.

[0035] Technical effects and advantages of the behavior analysis and correction system based on pet-perspective shooting data of the present invention:

[0036] The present invention is based on pet-perspective video, combined with acceleration data and gyroscope data, and can observe and analyze the behavior of the pet from the perspective of the pet in all directions; the pet-perspective video system can continuously collect new data as the pet's environment changes, and then analyze the pet's behavioral adaptability and changing patterns in various environments. Even if the environment is complex and changeable, the behavior analysis and correction strategies can still be adjusted based on the new data; and the video analysis model overcomes the jitter caused by video data shot from the pet's perspective, reduces image blur caused by rapid movement or jitter, improves video quality, and improves the accuracy of behavior recognition; it can also further analyze the pet personality tags of different pet individuals based on the collected pet-perspective video, and formulate behavior correction strategies that fit the individuals according to the pet personality tags, which is conducive to establishing a good pet behavior management model and achieving accurate and effective behavior correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a behavior analysis and correction system based on pet-perspective-based shooting data of the present invention;

[0038] Figure 2 It is a schematic diagram of facial key point features of the present invention;

[0039] Figure 3 A schematic diagram of the network architecture of the video analysis model of the present invention;

[0040] Figure 4 A schematic diagram of the pet behavior analysis and correction process of the present invention;

[0041] Figure 5 This is a schematic block diagram of the reinforcement learning algorithm structure of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example 1

[0044] See also Figure 1 As shown, the behavior analysis and correction system based on pet-perspective-shot data described in this embodiment includes:

[0045] Acquisition and processing module: collects pet's perspective video, acceleration data, gyroscope data and pet's behavior reaction time, and obtains video frame data, acceleration features and gyroscope features;

[0046] The pet-perspective video is acquired through a camera installed on the pet's back, and its format is RGB or RGB-D data. The pet-perspective video includes video data and its corresponding time data. The video data contains rich visual information about the pet's behavior, which helps to extract video frame data later and perform further feature extraction to identify the pet's various behavioral actions, providing a direct basis for pet behavior recognition. Video data of pet playmates can also be collected to better understand the pet's behavior in social scenarios, which can serve as an indirect reference for pet behavior recognition. Time data can facilitate the acquisition of video frame data later, so that the time analysis model can better learn the long-term temporal dependency of video analysis features, and improve the accuracy of behavior recognition results.

[0047] Both acceleration data and gyroscope data are collected corresponding to the time data of the video data shot from the pet's perspective. The acceleration data is obtained by collecting the accelerometer; the gyroscope data is obtained by collecting the gyroscope; collecting acceleration data and gyroscope data can accurately reflect the pet's movement state and the rotation information of the pet's body; combining these data with the video frame data can more comprehensively capture the details of the pet's behavior and actions, especially for some fast and complex actions, it can provide additional motion information support and improve the accuracy of behavior classification.

[0048] Methods for obtaining video frame data include:

[0049] The acquired video data is obtained in the order of time data to obtain the corresponding video frame sequence, and marked as {I z}(z=1,…,n), where I z represents the zth frame of video data; n is the number of video frames; obtain the pixel value corresponding to each frame of video data, and calculate the pixel difference δ between the zth frame and the z+1th frame of video data z :

[0050]

[0051] Where W, H and C are the width, height and number of channels of the video data respectively; XS z and XS z+1 are the pixel values ​​of the z-th frame and the z+1-th frame of video data respectively; x, y, and c are the horizontal position coordinates, vertical position coordinates, and channel number of the pixel of the video data respectively; this calculation is very important for applications such as video analysis, motion detection, and video compression, because it can help identify key frames or frames with large changes in the video, thereby optimizing the efficiency of video processing and analysis.

[0052] The pixel difference between the zth frame and the z+1th frame video data is compared with a preset pixel difference threshold, and when the pixel difference is not less than the preset pixel difference threshold, the z+1th frame is extracted to obtain video frame data.

[0053] Collecting video frame data can clearly show the details of the pet's own movements, which helps to accurately identify various types of pet behaviors, such as chasing, playing, resting, etc. It is also convenient for later analysis to obtain relevant data of the pet's playmates, which can serve as auxiliary support data for analyzing pet behavior.

[0054] Methods for obtaining acceleration characteristics and gyroscope characteristics include:

[0055] The time corresponding to each frame of video frame data is counted, and the average acceleration data and average gyroscope data corresponding to each frame of video frame data are calculated, and the calculated average acceleration data and average gyroscope data sequence are used as acceleration features and gyroscope features. Obtaining the acceleration features and gyroscope features corresponding to the video frame data can more comprehensively describe the pet behavior through motion information and provide the accuracy of behavior recognition.

[0056] The behavioral reaction time of a pet is the behavioral reaction time of the pet in G types of stimulation scenarios, which is obtained by measuring the pet after subjecting it to different types of stimulation. Collecting the behavioral reaction time of a pet in G types of stimulation scenarios can intuitively reflect the pet's sensitivity to external stimulation. By comparing the reaction times of different pets in the same stimulation scenarios, their personalities can be better divided. For example, when two pets face light stimulation of the same intensity, one pet can react in a short time, such as turning around, while the other pet reacts more slowly. This indicates that they have differences in their sensitivity to visual stimulation, which helps to label them with different pet personality labels later.

[0057] The methods for collecting the behavioral reaction time of pets in G types of stimulus scenarios include:

[0058] Set the types and parameters of G kinds of stimulation scenes. For the Vth type of stimulation scene among the G kinds of stimulation scenes, accurately record the time t1 when the stimulation starts, and also record the time t2 when the pet shows a behavioral response, and repeat the recording n times. t times, n t The average value of the difference between t1 and t2 is taken as the behavioral reaction time of the pet in the Vth type of stimulation scenario; wherein, when the acceleration data of the pet is higher than the preset acceleration threshold or the gyroscope data is higher than the preset gyroscope threshold, it indicates that the pet has a behavioral reaction.

[0059] The first analysis module: using the video frame data as the input of the target detection model, obtaining the probability (such as 0.2, 0.4 and 0.7, etc.) and the bounding box coordinates of whether the video frame data contains the pet playmate's facial area, the bounding box coordinates are usually expressed in the form of (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the bounding box, w is the width of the bounding box, and h is the height of the bounding box; comparing the probability with a preset probability threshold, when the probability is higher than the preset probability threshold, it means that the video feature contains a pet playmate, otherwise it does not contain a pet playmate; the video frame with a probability higher than the preset probability threshold is The data is marked as the first video data, and the remaining video frame data is marked as the second video data (for example, if the preset probability threshold is set to 0.5, the video frame data with a probability of 0.7 or higher than the preset probability threshold 0.5 is marked as the first video data, and the video frame data with a probability of 0.2 and 0.4 or lower than the preset probability threshold 0.5 is marked as the second video data), and the facial key point features of the pet playmate in the target area corresponding to the bounding box coordinates of the first video data are extracted, and the facial key point features are used as the input of the emotion recognition model to obtain the emotion recognition results of the pet playmate (such as relaxation and vigilance, etc.).

[0060] During the interaction between pets and their playmates, the emotion recognition results of the playmates can help infer the pet's behavioral intentions. For example, when the pet playmate shows friendly and happy emotions, the pet's approach and intimacy may be for social and playful purposes; and when the playmate shows aggressive emotions, the pet's avoidance or defensive behavior can be understood as the intention of self-protection. This can more accurately classify and identify the behavior of pets in social scenarios. There are phenomena such as behavioral associations and emotional contagion between pets and playmates. For example, during play, the happy emotions of pet playmates may be accompanied by mutual chasing, biting and other behaviors. Identifying the emotions of pet playmates can help us better associate and understand the relationship between these complex behaviors. If the playmate's emotions suddenly turn to anger, then the pet's emotions may also turn to anger, and the behavior pattern between the pet and the pet playmate may also change from playing to fighting. This emotional contagion effect and emotion-behavior association analysis can help to more comprehensively identify the behavior patterns of pets in complex social situations and help to more accurately identify the behavior of pets.

[0061] Reference Figure 2 The key features of the face include eye features, mouth features, and ear features. The eye features include the coordinates of the inner corner of the left eye, the outer corner of the left eye, the center of the pupil of the left eye, the inner corner of the right eye, the outer corner of the right eye, and the center of the pupil of the right eye; the mouth features include the coordinates of the left corner of the mouth, the midpoint of the upper lip, the midpoint of the lower lip, and the right corner of the mouth; the ear features include the coordinates of the left ear tip, the right ear tip, the left ear base, and the right ear base.

[0062] The training methods for the target detection model include:

[0063] A group of video training data is collected in advance, where the video training data includes video frame data and probabilities and bounding box coordinates corresponding to the video frame data.

[0064] Each set of video training data is used as the input of the target detection model. The target detection model uses the probability and bounding box coordinates corresponding to each set of video frame data as output, and uses the actual probability and bounding box coordinates corresponding to each set of video frame data as the prediction target; minimizing the sum of the prediction errors of all probabilities and bounding box coordinates is used as the training target; the target detection model is trained until the sum of the prediction errors reaches convergence, and the training is stopped; the target detection model is a deep neural network model. The deep neural network model is a CNN network model.

[0065] The target detection model loss function value is the mean square error.

[0066] Prediction error formula: u is the uth group of video frame data; A is the number of groups of video frame data; is the probability and bounding box coordinates corresponding to the u-th group of video frame data; z u is the actual probability and bounding box coordinates corresponding to the u-th group of video frame data.

[0067] The training method of the emotion recognition model can refer to the target detection model.

[0068] The second analysis module: taking the second video data as the input of the video analysis model to obtain the video analysis features; then taking the video analysis features as the input of the time series model to obtain the video features; the video analysis model is a multi-level CNN network integrating the convolution kernel size reduction strategy and the hole convolution module;

[0069] The training methods of the video analysis model and time series model can refer to the target detection model; among them, the video analysis model is a multi-layer CNN network model, refer to Figure 3 The multi-level CNN network model includes a de-jitter layer, an input layer, NH convolutional layers, a pooling layer, a fully connected layer, and an output layer; the de-jitter layer is composed of a dual-stream processing structure consisting of an optical flow estimation sub-network and a sensor fusion sub-module. The optical flow estimation sub-network adopts a lightweight FlowNet structure and calculates the first motion compensation amount M through the optical flow field between adjacent frames. optical :The first motion compensation corresponding to time k; the sensor fusion submodule inputs the acceleration data and gyroscope data into the LSTM network to predict the device motion trajectory to obtain the second motion compensation M sensor ; The de-jitter layer uses differentiable bilinear sampling combined with the first motion compensation amount and the second motion compensation amount to calculate the total compensation amount for motion compensation; such as the motion compensation amount Γ k =warp(V k ,M k ), where V k is the second video frame data; M k is the de-jitter layer compensation corresponding to time k, M k =αM optical +(1-α)M sensor ; Where α is the compensation coefficient, which can be dynamically adjusted according to the signal-to-noise ratio; M optical is the compensation matrix based on optical flow estimation; M sensorIt is a compensation matrix based on sensor data; the de-jitter layer can compensate for the rapid jitter of the video frame caused by the vigorous movement of the pet (such as running, jumping) or the shaking of the device; reduce the image blur caused by rapid movement and restore key details (such as facial expressions and body movements); ensure the alignment of features between adjacent frames to avoid timing analysis errors caused by jitter (such as misjudgment of action continuity); in the NH convolutional layers, the convolution kernel size of each convolutional layer decreases with the increase of the number of convolutional layers, among which the convolution kernel size of the first convolutional layer is the largest, and the convolution kernel size of the last convolutional layer is the smallest; the convolution kernel size of the first layer is large (such as 7×7), which can capture the overall shape of the pet and the large-amplitude movement characteristics of the pet. As the number of network layers deepens, the convolution kernel size gradually decreases (such as 3×3, 1×1), which can focus on the local details of the pet's body, such as head rotation, tail swinging and other subtle movements. This multi-scale design can adapt to the shooting data of pets at different distances and angles, and comprehensively extract behavioral features; a dilated convolution module is connected after each convolution layer; the dilated convolution module can expand the receptive field without increasing the number of parameters by inserting holes between the convolution kernel elements, so that the model can better understand the interaction between the pet and the surrounding environment, such as the behavioral feature capture of the pet in complex scenes such as turning around in a small space and shuttling obstacles. The output of the dilated convolution module connected after the nhth convolution layer is:

[0070]

[0071] Among them, ym nh [rt,rb] is the output feature map of the dilated convolution module connected after the nh-th convolution layer; [rt,rb] is the element position of the feature map, rt is the row index, and rb is the column index; xm nh is the input feature map of the hole convolution module connected after the nh-th convolution layer, that is, the output feature map of the nh-th convolution layer; kd is the hole rate; KL nh [cm,cn] is the element value corresponding to the position with row index cm and column index cn in the convolution kernel of the dilated convolution module connected after the nh-th convolution layer; kl nh ×kl nh is the size of the convolution kernel of the dilated convolution module connected after the nh-th convolution layer; the convolution kernel of the dilated convolution module is integrated into the deformable mechanism, and an offset prediction branch is added to the convolution kernel of each dilated convolution module to obtain the offset of each sampling position of the convolution kernel Δp = Φ(concatxm nh,S)); where S is the acceleration data; the offset is superimposed on the size of the convolution kernel of the original dilated convolution module to obtain the convolution kernel of the deformed dilated convolution module; the deformable mechanism of the dilated convolution module predicts an offset for each sampling position, so that the actual sampling position becomes the original position + predicted offset, so that the video analysis model can dynamically adjust the receptive field according to the input video image to adapt to geometric deformation and motion blur; when the sensor detects that the device is moving quickly (such as gyroscope data showing high-frequency jitter), the offset branch will predict a larger offset, so that the convolution kernel sampling position is adjusted in the opposite direction of the movement to compensate for the feature displacement caused by the blur; the offset branch can also automatically adjust the sampling position according to the image features, focusing on the key points of the deformation area (such as the local deformation caused by the pet's body (such as the swinging tail and the turning head)).

[0072] The time series model is an LSTM or Transformer network model. The behavior of pets is a process that changes dynamically over time. The time series model can handle this dynamicity well. By analyzing the input video analysis features in the time dimension, it can capture the characteristic changes of the start, duration, and end stages of the behavior, which helps to make more accurate identification of pet behavior in the later stage.

[0073] Behavior analysis module: The behavior analysis features are used as inputs to the behavior recognition model and the risk analysis model to obtain the pet's behavior recognition results (such as playing, eating, and aggressive behaviors) and risk assessment scores; the behavior analysis features include the emotion recognition results of the pet's playmates, video features, acceleration features, and gyroscope features;

[0074] Based on the comparison between the risk assessment score and the preset risk threshold, it can be determined whether the pet's behavior needs to be corrected. The specific process can be referred to Figure 4 ; The training methods of the behavior recognition model and the risk analysis model can refer to the video analysis model; by analyzing the pet's behavior recognition results and risk assessment scores, it can help the owner monitor the pet's health status and behavioral habits in real time, and warn of high-risk abnormal behaviors (such as biting furniture and abnormal postures for a long time).

[0075] The third analysis module: uses the pet's behavioral reaction time as the input of the personality analysis model to obtain the pet's personality label i (such as timid and lively, etc.), and defines a correction strategy library B for the pet's personality label i (such as the correction strategy for timid pets includes lower sound volume and vibration frequency. For lively pets, lower sound volume and vibration frequency may not be able to correct the pet's behavior, requiring higher sound volume and vibration frequency, or notifying the owner to comfort the pet).

[0076] The training method of the personality analysis model can refer to the video analysis model; pets with different pet personality labels may react very differently to the same correction strategy. By defining a correction strategy library for pet personality labels, the most suitable correction strategy can be selected according to the pet's personality characteristics, thereby significantly improving the effect of behavior correction.

[0077] The method of defining the correction strategy library B for the pet personality label i includes:

[0078] Corrective strategy library B contains the following corrective actions:

[0079] {b i11 ,b i12 ,…,b i1q ,…,b i1c ,b i21 ,b i22 ,…,b i2j ,…,b i2d ,b i31 ,b i32 ,…,b i3k ,…,b i3p};

[0080] Among them, b i1q represents the action in the sound prompt correction method corresponding to the pet personality tag i, q = [1, b], b is the number of actions in the sound prompt correction method corresponding to the pet personality tag i; b i2j represents the action in the device vibration correction method corresponding to the pet personality tag i, j = [1, d], d is the number of actions in the device vibration correction method corresponding to the pet personality tag i; b i3k represents the action in the smart notification method for the owner corresponding to the pet personality tag i, k = [1, p], where p is the number of actions in the smart notification method for the owner corresponding to the pet personality tag i;

[0081] In the sound prompt correction method, the sound prompt correction action is synthesized by different sound frequencies and sound volumes, and the parameter vector b is used. 1q =(f q ,v q ) indicates a sound prompt for corrective action, where f q is the frequency parameter of the qth sound prompt correction action, v q is the volume parameter of the qth sound prompt correction action; in the device vibration correction method, the device vibration correction action is synthesized by different vibration intensities and vibration frequencies, using the parameter vector b 1j =(E j ,F j ) represents the equipment vibration correction action, where E j F is the vibration intensity parameter of the jth device vibration correction action;j Vibration frequency parameter for the vibration correction action of the jth device.

[0082] Behavior correction module: The pet personality label i, behavior recognition results and risk assessment scores are used as inputs of the reinforcement learning algorithm. Through the dynamically updated Q-value function, combined with the exploration rate and the reward function obtained by weighting the behavior frequency and the stop frequency, the corresponding behavior correction strategy is generated from the correction strategy library B.

[0083] Ways to obtain behavior modification strategies include:

[0084] Reference Figure 5 , the behavior correction module is defined as the agent, the pet and its environment are defined as the environment, the pet's pet personality label, behavior recognition result and risk assessment score are defined as the environmental state, and the correction action corresponding to the pet personality label i in the correction strategy library B is defined as the action; and a reward function for evaluating the value of the action is also defined.

[0085] The reward function is defined as follows:

[0086] R t =ω1×l1+ω2×l2;

[0087]

[0088] Among them, R t is the reward function value corresponding to time step t; n1 is the number of times the pet performs the same behavior as the behavior recognition result within the preset time; n2 is the number of times the same behavior as the behavior recognition result occurs within the preset time after the behavior correction strategy is implemented; l1 is the behavior frequency; ω1 is the weight of the behavior frequency, which can be obtained by optimizing the natural inspiration optimization algorithm; m1 is the number of times the pet spontaneously stops the same behavior as the behavior recognition result within the preset time; m2 is the number of times the pet spontaneously stops the same behavior as the behavior recognition result within the preset time after the behavior correction strategy is implemented; l2 is the stop frequency; ω2 is the weight of the stop frequency, which can be obtained by optimizing the natural inspiration optimization algorithm;

[0089] Use the Q learning algorithm to learn the optimal strategy, initialize the Q value function as a table, the rows of the table correspond to different environmental states s, the columns correspond to different actions e, set the learning rate λ∈(0,1], discount factor δ∈[0,1] and exploration rate ∈∈[0,1]; at each time step t, observe the pet's current environmental state s t , according to the exploration rate ∈, randomly select an action e with probability ∈ t , select the action e' with the largest Q value in the current environment state with a probability of 1-∈ t; Execute the selected action, observe the pet's reaction, obtain new behavior recognition results and new risk assessment scores through the behavior analysis module, and thus obtain the new environment state s t+1 (The pet’s personality label generally remains unchanged). At the same time, the instant reward R is calculated based on the reward function t+1 ; Update the Q value function according to the update formula:

[0090]

[0091] Among them, Q'(s t ,e t ) is the updated Q value; Q(s t ,e t ) is the environmental state s t Next action t Q value; λ is the learning rate; R t+1 is the immediate reward; δ is the discount factor; a' is the next environment state s t+1 The best corrective action among all possible corrective actions; is the next environment state s t+1 The maximum Q value corresponding to all possible corrective actions e' under the current state is the maximum expected cumulative reward that can be obtained from all optional corrective actions after transferring from the current environment state to the next environment state.

[0092] The Q value function is updated repeatedly until the Q value converges to obtain the corresponding optimal Q value function. When the behavior analysis module detects that the pet's behavior needs to be corrected, the optimal Q value function is used to select the optimal action as the correction strategy from the correction strategy library corresponding to the pet's personality label i in the environmental state according to the environmental state and execute it.

[0093] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

[0094] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A behavior analysis and correction system based on pet-perspective shooting data, characterized in that: include: Acquisition and processing module: collects pet-perspective video, acceleration data, gyroscope data and behavioral reaction time, and obtains video frame data, acceleration features and gyroscope features through preprocessing; The first analysis module: uses the video frame data as the input of the target detection model to obtain the probability of containing the pet playmate's face and the bounding box coordinates; marks the video frame data with a probability higher than a preset probability threshold as the first data, marks the remaining video frame data as the second data, and extracts the key point features of the pet playmate's face in the first data; The facial key point features are used as the input of the emotion recognition model to obtain the emotion recognition results of the pet playmate; The second analysis module uses the second data as an input of the video analysis model to obtain video analysis features; Using the video analysis features as the input of the time series model to obtain video features; The video analysis model is a multi-layer CNN network, which includes a de-jitter layer and NH convolutional layers. The de-jitter layer consists of a dual-stream processing structure consisting of an optical flow estimation sub-network and a sensor fusion sub-module. The size of the convolution kernel of each NH convolutional layer decreases as the number of convolutional layers increases. Each convolutional layer is connected to a dilated convolution module in which the convolution kernel is integrated into a deformable mechanism. Behavior analysis module: The behavior analysis features are used as inputs of the behavior recognition model and the risk analysis model to obtain the corresponding pet behavior recognition results and risk assessment scores; the behavior analysis features include the emotion recognition results, video features, acceleration features, and gyroscope features of the pet playmate; The third analysis module: taking the behavior reaction time as the input of the personality analysis model, obtaining the pet personality label i, and defining the correction strategy library B for the pet personality label i; Behavior correction module: The pet personality label i, behavior recognition results and risk assessment scores are used as inputs to the reinforcement learning algorithm to obtain the behavior correction strategy.

2. The behavior analysis and correction system based on pet perspective shooting data according to claim 1, characterized in that: The video analysis model is a multi-level CNN network model, which includes a de-jitter layer, an input layer, NH convolutional layers, a pooling layer, a fully connected layer and an output layer; the de-jitter layer is composed of a dual-stream processing structure consisting of an optical flow estimation sub-network and a sensor fusion sub-module, wherein the optical flow estimation sub-network adopts a lightweight FlowNet structure to calculate the first motion compensation amount through the optical flow field between adjacent frames: the sensor fusion sub-module inputs acceleration data and gyroscope data into the LSTM network to predict the motion trajectory of the device to obtain the second motion compensation amount; the de-jitter layer adopts differentiable bilinear sampling combined with the first motion compensation amount and the second motion compensation amount to calculate the total compensation amount for motion compensation; in the NH convolutional layers, the convolution kernel size of each convolutional layer is The size decreases with the increase of the number of convolution layers, among which the convolution kernel size of the first convolution layer is the largest, and the convolution kernel size of the NHth convolution layer is the smallest. A dilated convolution module is connected to each convolution layer; the output of the dilated convolution module connected to the nhth convolution layer is obtained by performing a convolution operation on the corresponding input, nh=1,…,NH; the convolution operation is used to perform a weighted summation on each element position of the corresponding input; during the convolution operation, the interval between elements is controlled by the dilation rate; the convolution kernel of the dilated convolution module is integrated into the deformable mechanism, and an offset prediction branch is added to the convolution kernel of each dilated convolution module. By obtaining the offset of each sampling position of the convolution kernel, the offset is superimposed on the size of the convolution kernel of the original dilated convolution module to obtain the deformed convolution kernel of the dilated convolution module.

3. The behavior analysis and correction system based on pet perspective shooting data according to claim 1, characterized in that: The correction actions of the correction strategy library B include: actions in the sound prompt correction method, actions in the device vibration correction method and actions in the owner intelligent notification method; in the sound prompt correction method, the sound prompt correction action is synthesized by different sound frequencies and sound volumes; in the device vibration correction method, the device vibration correction action is synthesized by different vibration intensities and vibration frequencies.

4. The behavior analysis and correction system based on pet perspective shooting data according to claim 3 is characterized in that: The method for obtaining a behavior correction strategy comprises: The behavior correction module is defined as an agent, the pet and its environment are defined as the environment, the pet's pet personality tag, behavior recognition result and risk assessment score are defined as the environment state, and the correction action corresponding to the pet personality tag in the correction strategy library B is defined as an action; and a reward function for evaluating the value of an action is also defined; Use the Q learning algorithm to learn the optimal strategy, initialize the Q value function as a table, the rows of the table correspond to different environmental states s, the columns correspond to different actions e, set the learning rate λ, discount factor δ and exploration rate ∈; at each time step t, observe the pet’s current environmental state s t , according to the exploration rate ∈, randomly select an action with probability ∈, and select the action with the largest Q value in the current environment state with probability 1-∈; execute the selected action, observe the pet's reaction, obtain new behavior recognition results and new risk assessment scores through the behavior analysis module, and obtain the new environment state s t+1 ; At the same time, the immediate reward R is calculated according to the reward function t+1 ; Repeat the Q value function until the Q value converges to obtain the corresponding optimal Q value function. When the behavior analysis module detects that the pet's behavior needs to be corrected, the optimal Q value function is used to select the optimal action as the correction strategy from the correction strategy library corresponding to the pet's personality label i in the environmental state according to the environmental state and execute it; The update of the Q value function is specifically as follows: The updated Q value is the sum of the current Q value and the correction term. The correction term is the product of the learning rate and the difference. The difference is the immediate reward plus the discounted maximum Q value of the next state, minus the current Q value.

5. The behavior analysis and correction system based on pet perspective shooting data according to claim 4, characterized in that: The reward function is a weighted combination of behavior frequency and stop frequency, wherein the behavior frequency is the ratio of the difference in the number of times the pet performs a specific behavior before and after the implementation of the behavior correction strategy to the number of times the pet performs the specific behavior before the implementation of the behavior correction strategy; the stop frequency is the ratio of the difference in the number of times the pet spontaneously stops a specific behavior before and after the implementation of the behavior correction strategy to the number of times the pet spontaneously stops the specific behavior before the implementation of the behavior correction strategy.

6. The behavior analysis and correction system based on pet perspective shooting data according to claim 1, characterized in that: The behavior reaction time of the pet is the behavior reaction time of the pet in G types of stimulation scenes. The method of collecting the behavior reaction time of the pet in G types of stimulation scenes includes: Set the types and parameters of G kinds of stimulation scenes. For the Vth type of stimulation scene among the G kinds of stimulation scenes, record the time t1 when the stimulation starts, and also record the time t2 when the pet shows a behavioral response, and repeat the recording n times. t times, n t The average value of the difference between t1 and t2 is taken as the behavioral reaction time of the pet in the Vth type of stimulation scenario; wherein, when the acceleration data of the pet is higher than the preset acceleration threshold or the gyroscope data is higher than the preset gyroscope threshold, it indicates that the pet has a behavioral reaction.

7. The behavior analysis and correction system based on pet perspective shooting data according to claim 1, characterized in that: The pet perspective video includes video data and its corresponding time data; Methods for obtaining video frame data include: The acquired video data is obtained in the order of time data to obtain the corresponding video frame sequence. For each frame of video data, all its pixels are traversed, and the square of the difference between the corresponding pixels of the t-th frame and the t+1-th frame is calculated. The square of the difference of all pixel points is summed, and the sum is divided by the product of the width, height and number of channels of the video data to obtain the pixel difference between the t-th frame and the t+1-th frame of video data; The pixel difference between the t-th frame and the t+1-th frame video data is compared with a preset pixel difference threshold, and when the pixel difference is not less than the preset pixel difference threshold, the t+1-th frame is extracted to obtain video frame data.

8. The behavior analysis and correction system based on pet perspective shooting data according to claim 7, characterized in that: Methods for obtaining acceleration characteristics and gyroscope characteristics include: The time corresponding to each frame of video frame data in the video frame data is counted, the average acceleration data and the average gyroscope data corresponding to each frame of video frame data are calculated, and the calculated average acceleration data and average gyroscope data sequences are used as acceleration features and gyroscope features.

9. The behavior analysis and correction system based on pet perspective shooting data according to claim 1, characterized in that: In the behavior analysis module, whether the pet behavior needs to be corrected is determined based on the comparison result between the risk assessment score and the preset risk threshold. When the risk assessment score exceeds the preset risk threshold, it is determined that the pet behavior needs to be corrected, otherwise the pet behavior does not need to be corrected.

10. The behavior analysis and correction system based on pet perspective shooting data according to claim 1, characterized in that: The facial key point features include eye features, mouth features, and ear features. The eye features include the coordinates of the inner corner of the left eye, the outer corner of the left eye, the center of the pupil of the left eye, the inner corner of the right eye, the outer corner of the right eye, and the center of the pupil of the right eye; the mouth features include the coordinates of the left corner of the mouth, the midpoint of the upper lip, the midpoint of the lower lip, and the coordinates of the right corner of the mouth; the ear features include the coordinates of the left ear tip, the right ear tip, the left ear base, and the right ear base.

Images