Data processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a to-be-detected image from to-be-detected video data; inputting the to-be-detected image into a target recognition model to recognize the to-be-detected image to obtain object information contained in the to-be-detected image; under the condition that the object information represents that the pet exists in the to-be-detected image, inputting the to-be-detected image into a target classification model to obtain a character action of a character included in the to-be-detected image; and based on the object information and / or the character action, determining an interaction behavior between a character and a pet in the to-be-detected image. By setting the target recognition model and the target classification model, the interactive behavior between the figure and the pet can be detected in combination with object recognition and figure action recognition, and the interactive behavior is detected in combination with the figure action, so that the detection accuracy can be improved to a certain extent.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device, electronic device and readable storage medium. Background Art

[0002] With the development of society, pets are becoming more and more common in people's daily lives. People interact with pets in different ways in different occasions. For example, people feed their pets indoors and walk their pets in public outdoors.

[0003] As the number of pets increases, people have a lot of interactions with their pets, and some people have illegal interactions with their pets (for example, walking their pets without a leash). Therefore, how to detect the interaction between people and pets has become an urgent problem that needs to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data processing method, device, electronic device and readable storage medium to detect the interactive behavior between a person and a pet. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a data processing method is provided, the method comprising:

[0006] Acquire an image to be detected from video data to be detected;

[0007] Inputting the image to be detected into the target recognition model to recognize the image to be detected and obtain the object information contained in the image to be detected;

[0008] In a case where the object information indicates that a pet exists in the image to be detected, the image to be detected is input into a target classification model to obtain a person action of a person contained in the image to be detected;

[0009] Based on the object information and / or the person's action, the interactive behavior between the person and the pet in the image to be detected is determined.

[0010] Optionally, the target model is trained by the following steps:

[0011] Get multiple sample images and sample labels;

[0012] Inputting the plurality of sample images into an initial detection model to obtain a sample recognition result corresponding to each of the sample images;

[0013] Based on the sample recognition result and the sample label, the model parameters of the initial detection model are adjusted until a training stop condition is reached, and the current initial detection model is determined as a target model;

[0014] The sample labels are used to characterize the object information contained in each of the sample images, and the target model is the target recognition model; or the sample labels are used to characterize the character actions of the characters contained in each of the sample images, and the target model is the target classification model.

[0015] Optionally, when the sample label is used to characterize a person action of a person included in each of the sample images, obtaining a sample recognition result corresponding to each of the sample images includes:

[0016] Obtaining key points of characters in each of the sample images;

[0017] Generate a character feature vector based on the character key points;

[0018] The character action is determined based on the character feature vector.

[0019] Optionally, when the sample label is used to characterize object information contained in each of the sample images, the initial detection model at least includes a semantic segmentation model.

[0020] Optionally, the object information includes a person position, a pet position, and a traction component position; and determining the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person action includes:

[0021] In the case where the distance between the pet position and the traction component position is not greater than a preset distance threshold, if the human action belongs to a preset detection action, determining that the interaction behavior between the human and the pet in the image to be detected belongs to a preset detection behavior;

[0022] Or, when the distance between the pet position and the traction component position is greater than the preset distance threshold, it is determined that the interactive behavior between the person and the pet in the image to be detected belongs to the preset detection behavior.

[0023] Optionally, acquiring the image to be detected from the video data to be detected includes:

[0024] Acquire a plurality of continuous video frames from the video data to be detected as a plurality of images to be detected;

[0025] The method further comprises:

[0026] In a case where the interactive behavior belongs to a preset detection behavior, determining the image to be detected as a target image;

[0027] When the number of the target images existing in the multiple images to be detected is not less than a preset number threshold, it is determined that the preset detection behavior exists in the video data to be detected.

[0028] In a second aspect of the present invention, there is also provided a data processing device, the device comprising:

[0029] A first acquisition module, used for acquiring an image to be detected from video data to be detected;

[0030] A first input module, used for inputting the image to be detected into the target recognition model to recognize the image to be detected and obtain the object information contained in the image to be detected;

[0031] A second input module is used for inputting the image to be detected into a target classification model to obtain the person action of the person contained in the image to be detected when the object information indicates that there is a pet in the image to be detected;

[0032] The behavior determination module is used to determine the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person's action.

[0033] Optionally, the device further comprises:

[0034] A second acquisition module is used to acquire multiple sample images and sample labels;

[0035] A third input module, used for inputting the plurality of sample images into an initial detection model to obtain a sample recognition result corresponding to each of the sample images;

[0036] A parameter adjustment module, used to adjust the model parameters of the initial detection model based on the sample recognition result and the sample label, until the training stop condition is reached, and the current initial detection model is determined as the target model;

[0037] The sample labels are used to characterize the object information contained in each of the sample images, and the target model is the target recognition model; or the sample labels are used to characterize the character actions of the characters contained in each of the sample images, and the target model is the target classification model.

[0038] Optionally, when the sample labels are used to characterize the actions of the characters contained in each of the sample images, the third input module includes:

[0039] A third acquisition submodule is used to acquire key points of characters in each of the sample images;

[0040] A vector generation submodule, used to generate a character feature vector based on the character key points;

[0041] The action determination submodule is used to determine the character action based on the character feature vector.

[0042] Optionally, when the sample label is used to characterize object information contained in each of the sample images, the initial detection model at least includes a semantic segmentation model.

[0043] Optionally, the behavior determination module includes:

[0044] A first determination submodule is used for determining that the interaction between the person and the pet in the image to be detected belongs to the preset detection behavior if the distance between the pet position and the traction component position is not greater than a preset distance threshold and if the person action belongs to the preset detection action;

[0045] Or, a second determination submodule is used to determine that the interactive behavior between the person and the pet in the image to be detected belongs to the preset detection behavior when the distance between the pet position and the traction component position is greater than the preset distance threshold.

[0046] Optionally, the first acquisition module is specifically configured to:

[0047] Acquire a plurality of continuous video frames from the video data to be detected as a plurality of images to be detected;

[0048] The device also includes:

[0049] A third determination module, configured to determine the image to be detected as a target image when the interactive behavior belongs to a preset detection behavior;

[0050] The fourth determination module is used to determine that the video data to be detected has the preset detection behavior when the number of the target images existing in the multiple images to be detected is not less than a preset number threshold.

[0051] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0052] Memory, used to store computer programs;

[0053] The processor is used to implement the method described in the first aspect when executing the program stored in the memory.

[0054] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect above.

[0055] In a fifth aspect of the implementation of the present invention, there is also provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect above.

[0056] The data processing method provided by the embodiment of the present invention obtains an image to be detected from the video data to be detected; inputs the image to be detected into a target recognition model to identify the image to be detected and obtain the object information contained in the image to be detected; when the object information indicates that there is a pet in the image to be detected, inputs the image to be detected into a target classification model to obtain the person action of the person contained in the image to be detected; based on the object information and / or the person action, determines the interactive behavior between the person and the pet in the image to be detected. In the embodiment of the present invention, by setting a target recognition model and a target classification model, the interactive behavior between the person and the pet can be detected in combination with object recognition and person action recognition, and by detecting the interactive behavior in combination with the person action, the accuracy of the detection can be improved to a certain extent, and the effect of the interactive behavior detection can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0058] Figure 1 is a flow chart of steps of a data processing method in an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a scenario in an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a data processing method according to an embodiment of the present invention;

[0061] Figure 4 is a flow chart of another data processing method in an embodiment of the present invention;

[0062] Figure 5 is a structural schematic diagram of a data processing device in an embodiment of the present invention;

[0063] Figure 6 Schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Figure 1 is a flow chart of steps of a data processing method in an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0066] Step 101: Acquire an image to be detected from video data to be detected.

[0067] Step 102: input the image to be detected into a target recognition model to recognize the image to be detected and obtain object information contained in the image to be detected.

[0068] Step 103: When the object information indicates that a pet exists in the image to be detected, the image to be detected is input into a target classification model to obtain the action of the person contained in the image to be detected.

[0069] Step 104: Determine the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person's action.

[0070] With respect to the above steps 101 to 104, the embodiments of the present invention can be applied to any computer device with computing processing capabilities, and can be applied to the fields of artificial intelligence, computer vision, and video surveillance. Specifically, since many pet-raising families have installed home surveillance, and many public places are generally equipped with surveillance cameras, the interaction between people and pets can usually be monitored through surveillance videos. Among them, the above pets can be cats, dogs, monkeys, rabbits, etc., and can be set according to actual needs. The embodiments of the present invention do not limit this.

[0071] Among them, the video data to be detected can be indoor monitoring data of a pet-raising household or monitoring data of a public place, and can be set according to actual monitoring needs, which is not limited by the embodiment of the present invention. Specifically, the video data to be detected can be actively acquired through a camera interface, or uploaded by a user or actively sent by a camera. Furthermore, the video data to be detected can be acquired in real time or periodically according to a preset period, which is not limited by the embodiment of the present invention.

[0072] Specifically, one or more video frames can be randomly extracted from the video data to be detected as the images to be detected, or, by receiving user instructions, video frames at a certain moment or a certain time period can be selected as the images to be detected. This can be set according to actual needs, and the embodiments of the present invention do not limit this.

[0073] Among them, the above-mentioned target recognition model and target classification model can be pre-trained. Specifically, the target recognition model is used to identify the object information contained in the image, and the above-mentioned object information may include the position of the person, the position of the pet, and the position of the traction component, etc., and of course, it may also include information of other objects that need to be monitored. Among them, the above-mentioned traction component refers to a device used to pull a pet, which may be a traction rope. Specifically, the target classification model is used to identify human actions, which may include pet walking actions, beating actions, or feeding actions, etc.

[0074] Furthermore, the target recognition model can be obtained by training based on the target detection model and / or the semantic segmentation model. Accordingly, the target classification model can be obtained by training based on a binary classification model or a multi-classification model.

[0075] Specifically, the embodiment of the present invention can input the image to be detected as input data into the target recognition model, and the target recognition model can recognize the image to be detected and identify the object information contained therein, and then obtain the object information contained in the image to be detected by obtaining the output data of the target recognition model. Specifically, when the object information includes the position of a pet, it indicates that there is a pet in the image to be detected, and then the human action can be further detected at this time, so the image to be detected can be input again as input data into the target classification model, and the target classification model recognizes the person in the image to be detected and identifies the human action.

[0076] Specifically, when a person interacts with a pet, the person often performs corresponding actions, for example, when walking a dog on a leash, the person often extends his arms. Therefore, the embodiment of the present invention can determine the interactive behavior of the person and the pet in the image to be detected based on the object information and / or the person's actions. Specifically, the distance between the person and the pet can be determined based on the person's position and the pet's position in the object information, and further the interactive behavior between the person and the pet can be determined based on the distance between the person and the pet and the person's actions.

[0077] Furthermore, after determining the interactive behavior, the embodiment of the present invention can generate corresponding output information based on the interactive behavior and actual monitoring needs, for example, it can output prompt information, and the prompt information is used to indicate that the image to be detected has interactive behavior. The above prompt information can be in the form of a prompt light, a string, etc., and the embodiment of the present invention does not limit this.

[0078] In summary, in an embodiment of the present invention, an image to be detected is obtained from video data to be detected; the image to be detected is input into a target recognition model to identify the image to be detected and obtain object information contained in the image to be detected; when the object information indicates that there is a pet in the image to be detected, the image to be detected is input into a target classification model to obtain the person action of the person contained in the image to be detected; based on the object information and / or the person action, the interactive behavior between the person and the pet in the image to be detected is determined. In an embodiment of the present invention, by setting a target recognition model and a target classification model, the interactive behavior between the person and the pet can be detected in combination with object recognition and person action recognition, and by detecting the interactive behavior in combination with person actions, the accuracy of the detection can be improved to a certain extent, and the effect of the interactive behavior detection can be ensured.

[0079] Optionally, in the embodiment of the present invention, the target model can be obtained by training through the following steps:

[0080] S21. Obtain multiple sample images and sample labels.

[0081] S22: Input the plurality of sample images into an initial detection model to obtain a sample recognition result corresponding to each of the sample images.

[0082] S23. Based on the sample recognition result and the sample label, the model parameters of the initial detection model are adjusted until the training stop condition is reached, and the current initial detection model is determined as the target model.

[0083] The sample labels are used to characterize the object information contained in each of the sample images, and the target model is the target recognition model; or the sample labels are used to characterize the character actions of the characters contained in each of the sample images, and the target model is the target classification model.

[0084] The sample images may be pre-collected, or may be obtained from historical data collected through the Internet or a camera. Different sample images may include different human-pet interaction behaviors, and different sample images may also include different people and pets. The number of the sample images may be set voluntarily, and may be 100, 1000, 500, etc., and the embodiment of the present invention does not limit this.

[0085] The sample labels are used to characterize the object information contained in each sample image, and accordingly, the target model is a target recognition model. Specifically, the sample labels may include people and pets marked with rectangular frames, and may also include traction components marked point by point to achieve pixel-level annotation of traction components. Specifically, the sample labels may be manually annotated, or may be annotated using an annotation tool or a pixel segmentation model, and the embodiments of the present invention do not limit this.

[0086] Furthermore, in the case where the sample label is used to characterize the object information contained in each of the sample images, the initial detection model at least includes a semantic segmentation model.

[0087] Specifically, in the case where the sample labels are used to characterize the object information contained in each sample image, the above-mentioned initial detection model may include a semantic segmentation model and a target detection model. The semantic segmentation model can be used to perform pixel-level positioning of the traction component, and the target detection model can be used to identify and locate people and pets. In one case, the above-mentioned initial detection model can be a model that integrates target detection and semantic segmentation functions (You Only Look Once V8, YoloV8). YoloV8 is an algorithm framework created and maintained by Ultralytics that combines algorithms such as target detection, key point detection, and instance segmentation. It evolved from the single-stage target detection network Yolo, has relatively outstanding speed and accuracy performance, and is widely used in the field of target detection. Then, the embodiment of the present invention can train the initial detection model that at least includes a semantic segmentation model by using sample images and sample labels that characterize object information, and can improve the accuracy of the target model in locating and identifying objects.

[0088] Wherein, when the above-mentioned sample labels are used to characterize the character actions of the characters contained in each of the sample images, the above-mentioned target model is a target classification model. Accordingly, the above-mentioned initial detection model can be a binary classification model or a multi-classification model. Specifically, in one case, the above-mentioned initial detection model can be a bilinear image classification network (Bilinear Convolutional Neural Networks, Bilinear CNN), which is a fine-grained image classification model that examines the interaction between different dimensions by calculating the outer product of the convolution description vector, thereby capturing the relationship between different semantic features of the input image. Taking the dog walking behavior detection as an example, images containing human dog walking actions and non-dog walking actions can be collected as sample images. At the same time, the human body key points of the pedestrians in the picture are annotated, and the small picture formed by deducting the pedestrian area image in the complete picture is obtained as a key point annotated image, and the image is classified into two categories of dog walking action and non-dog walking action as the sample label of the sample image.

[0089] Specifically, the sample image can be input into the initial detection model, and the sample recognition results corresponding to each sample image can be obtained by reading the output data of the initial detection model. Then, based on the sample recognition results and the similarity or difference of the sample labels, the model parameters of the initial detection model can be adjusted until the training stop condition is reached, and the current initial detection model is determined as the target model.

[0090] Among them, the above-mentioned training stop condition can be that the number of model parameter adjustments reaches a preset number threshold, or that the preset loss function of the model reaches a preset loss threshold, etc. It can be set according to actual needs, and the embodiment of the present invention does not limit this.

[0091] In an embodiment of the present invention, a plurality of sample images and sample labels are obtained; the plurality of sample images are input into an initial detection model to obtain sample recognition results corresponding to each of the sample images; based on the sample recognition results and the sample labels, the model parameters of the initial detection model are adjusted until the training stop condition is reached, and the current initial detection model is determined as the target model; wherein the sample labels are used to characterize the object information contained in each of the sample images, and the target model is the target recognition model; or, the sample labels are used to characterize the character actions of the characters contained in each of the sample images, and the target model is the target classification model. In this way, the target recognition model and the target classification model are obtained by pre-training with sample images and sample labels, which can ensure the accuracy of model recognition, thereby further ensuring the accuracy of interactive behavior recognition.

[0092] Optionally, in the case where the sample labels are used to characterize the actions of the characters contained in the sample images, the operation of obtaining the sample recognition results corresponding to the sample images may specifically include:

[0093] S31, obtaining key points of characters in each of the sample images.

[0094] S32: Generate a character feature vector based on the character key points.

[0095] S33: Determine the character action based on the character feature vector.

[0096] The above-mentioned key points of the characters refer to the key points of the human body, which may include key parts of the human body such as joints and facial features. Specifically, the key points of the characters in the sample image may be obtained through the key point detection layer in the initial detection model. The above-mentioned key point detection layer may be formed by YoloV8, which may obtain the key points of the characters according to the postures of the characters in the sample image.

[0097] Furthermore, after obtaining the key points of the character, a character feature vector can be generated based on the key points of the character. Specifically, the coordinates of each key point of the character can be determined based on the key points of the character, and then the key point coordinates are combined to obtain a coordinate sequence, and the coordinate sequence is used as the character feature vector. It can be understood that the coordinates of the key points of the character corresponding to different character actions are different, and then the character feature vectors are also different, so the character action can be determined based on the character feature vector.

[0098] Furthermore, after obtaining the character feature vector, the character action can be classified based on the character feature vector using the classification layer in the initial detection model to determine the character action. Specifically, the classification layer can be formed using Bilinear CNN, so that the initial detection model can identify the character action based on the character key points through the key point detection layer and the classification layer. Exemplarily, taking dog walking behavior detection as an example, the feature vector composed of key points can be classified into two categories, one category represents dog walking actions and the other represents non-dog walking actions. In the embodiment of the present invention, a binary classification model is used as the classification layer to perform classification.

[0099] Taking dog walking behavior detection as an example, in the related technology, one method is to detect three types of targets: pedestrians, dogs, and leashes, and set a distance threshold to determine whether a person is walking the dog with a leash by judging the intersection of people, dogs, and leashes. Because the leash target is slender and small in the monitoring screen, it is easy to miss the leash using target detection. In addition, using a fixed distance threshold is difficult to cope with leash states of different lengths. There is another way to detect three types of targets: pedestrians, dogs, and leashes, and classify pedestrian images into two categories of targets: whether they are with a leash or not, and whether they are with a leash or not. Use a classification model to distinguish the two categories. This method is more robust than the first method, but it also has the problem of classification errors.

[0100] In the embodiment of the present invention, the key points of the characters in each sample image are obtained; the character feature vector is generated based on the key points of the characters; and the character action is determined based on the character feature vector. Introducing the key points of the human body to judge the character action is more accurate than the simple distance threshold judgment. For example, when walking a dog with a leash, the dog walker can be judged without holding the leash in his hand. This further improves the accuracy.

[0101] Optionally, the object information includes a person position, a pet position, and a traction component position; and determining the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person action includes:

[0102] S41, when the distance between the pet position and the traction component position is not greater than a preset distance threshold, if the human action belongs to a preset detection action, determining that the interaction behavior between the human and the pet in the image to be detected belongs to the preset detection behavior.

[0103] Or, S42, when the distance between the pet position and the traction component position is greater than the preset distance threshold, determine that the interactive behavior between the person and the pet in the image to be detected belongs to the preset detection behavior.

[0104] Among them, the above-mentioned preset detection behavior can be pre-set, and can be a certain interactive behavior that needs to be detected, for example, it can be the behavior of walking a pet without a leash or any other interactive behavior that needs to be detected. The above-mentioned preset detection behavior can be set according to actual needs, and the embodiments of the present invention do not limit this. Accordingly, the above-mentioned preset detection action can be a human action corresponding to the preset detection behavior, and can be a non-pet walking action. Among them, the above-mentioned preset distance threshold can be a pixel distance or a corresponding real distance value. Accordingly, the distance between the above-mentioned pet position and the traction component position can be a pixel distance or a corresponding real distance, and can be set according to the resolution of the video data to be processed, and the embodiments of the present invention do not limit this.

[0105] Specifically, the pixel area of ​​the leash and the positions of the person and the pet in the image may be located, and the minimum enclosing rectangular frame of the leash pixels may be obtained as the position of the leash.

[0106] Furthermore, when the distance between the pet's position and the traction component's position is not greater than a preset distance threshold, it indicates that there is a traction component near the pet. In this case, if the character's action belongs to a preset detection action, it can indicate that the interaction between the character and the pet belongs to a preset detection action.

[0107] Accordingly, when the distance between the pet's position and the traction component's position is greater than the preset distance threshold, it indicates that there is no traction rope near the pet. At this time, the pet is often in a wandering state or an untraction state, which is not compliant. At this time, it can be determined that the interactive behavior belongs to the preset detection behavior.

[0108] Optionally, when the distance between the pet position and the traction component position is not greater than the preset distance threshold, it can be further determined whether the distance between the traction component position and the person position is not greater than the preset distance threshold. If so, it indicates that the pet is being pulled by the person. Correspondingly, if not, the pet is often in a wandering state or an unpulled state, which is not compliant, and it can be determined that the interactive behavior belongs to the preset detection behavior.

[0109] For example, if a pet is a dog, after the dog target is detected, if there is a leash near the dog and there is a pedestrian walking the dog near the leash, the dog is in a normal compliance state. If there is no leash near the dog target, or there is no pedestrian walking the dog connected to the leash, the dog is in a wandering state, and no one is using the leash to lead the dog, which is a violation.

[0110] In the embodiment of the present invention, when the distance between the pet position and the traction component position is not greater than the preset distance threshold, if the person's action belongs to the preset detection action, it is determined that the interaction between the person and the pet in the image to be detected belongs to the preset detection behavior; or, when the distance between the pet position and the traction component position is greater than the preset distance threshold, it is determined that the interaction between the person and the pet in the image to be detected belongs to the preset detection behavior. In this way, the interaction between the person and the pet can be identified in combination with the distance between the pet and the traction component and the person's action, and the accuracy of the identification can be guaranteed to a certain extent.

[0111] Optionally, the operation of acquiring the image to be detected from the video data to be detected may specifically include:

[0112] S51 . Acquire a plurality of continuous video frames from the video data to be detected as a plurality of images to be detected.

[0113] The embodiment of the present invention may further include:

[0114] S52: When the interactive behavior belongs to a preset detection behavior, determine the image to be detected as a target image.

[0115] S53: When the number of the target images in the multiple images to be detected is not less than a preset number threshold, determine that the video data to be detected has the preset detection behavior.

[0116] Among them, multiple continuous video frames can be randomly obtained from the video data to be detected as multiple images to be detected, which can be 10, 20, 15, etc., and the embodiment of the present invention does not limit this.

[0117] Among them, the above-mentioned preset number threshold can be pre-set and can be set according to the number of images to be detected. For example, when the number of images to be detected is 10, the preset number threshold can be set to 5, 6, 4, etc., and the embodiment of the present invention does not limit this. Further, in the case where the interactive behavior belongs to the preset detection behavior, it indicates that the preset detection behavior exists in the image to be detected, and the image to be detected can be determined as the target image at this time. Correspondingly, when the number of target images existing in multiple images to be detected is not less than the preset number threshold, it indicates that there is a preset detection behavior in a certain continuous time period in the video data to be detected, which can greatly reduce the problem of recognition error in an image to be detected, and it can be determined that the video data to be detected has the above-mentioned preset detection behavior. At this time, a prompt message can be output to facilitate timely processing by relevant personnel.

[0118] Optionally, since there may be multiple people or multiple pets in the image, the above step S53 can track each person, leash and pet in the multi-frame image by a multi-target tracking algorithm (e.g., Bytetrack). Bytetrack is a simple and efficient multi-target tracking method, which improves the irrationality of the previous tracking algorithm discarding low-score detection frames, processes high-score frames and low-score frames separately, and uses the similarity between the trajectories of the low-score detection frames to mine potential objects in the low-score frames, thereby improving the overall tracking performance. In this way, misjudgment caused by a single frame can be avoided, and the stability of the system can be improved.

[0119] In an embodiment of the present invention, multiple continuous video frames are obtained from the video data to be detected as multiple images to be detected; when the interactive behavior belongs to a preset detection behavior, the image to be detected is determined as a target image; when the number of the target images existing in the multiple images to be detected is not less than a preset number threshold, it is determined that the video data to be detected has the preset detection behavior. By obtaining multiple images to be detected and judging by the preset number threshold, the misidentification caused by a single frame can be reduced, and the accuracy of behavior recognition can be further improved.

[0120] Figure 2 is a schematic diagram of a scenario in an embodiment of the present invention, such as Figure 2As shown in the figure, with the improvement of people's living standards, more and more people will keep pet dogs, and dog walking is an indispensable part of keeping pet dogs. When walking a dog, there are corresponding laws and regulations to be followed, especially the use of a leash for walking a dog. my country has also issued relevant animal epidemic prevention laws and regulations, and it is illegal to walk a dog without a leash. However, in public places, the phenomenon of walking a dog without a leash still occurs. Public places are generally equipped with surveillance cameras. If the behavior of illegal dog walking can be captured in time through the surveillance camera screen, it is of great significance to maintain public safety. In related technologies, the detection of illegal dog walking behavior is largely concentrated on the discrimination of single-frame images, and the target detection method is used to detect three types of targets: pedestrians, dogs, and leash. Because the target size of the leash is special, it is easy to miss detection. Whether the action of walking a dog with a leash is judged by the distance threshold between targets or image classification, the leash may intersect with the dog and the person, but there is no dog walking action, which is easy to misjudge and the detection efficiency is low.

[0121] Figure 3 FIG. 1 is a flow chart of a data processing method in an embodiment of the present invention. Figure 3 As shown, the embodiment of the present invention can use the segmentation and target detection model to detect the monitoring screen, obtain pedestrians, dogs, and traction ropes, and then when it is detected that there is no dog, it indicates that the monitoring screen is in a normal state at this time. Accordingly, when the presence of a dog is detected, the distance between the pedestrian, the dog, and the traction rope can be calculated, and it can be determined whether there is a traction rope near the dog. If not, it can be directly determined that it is an illegal dog walking state at this time, and there is an illegal state in the monitoring screen at this time. Correspondingly, if there is a traction rope near the dog, it can be further determined whether there is a person near the traction rope. If not, it can be directly determined that it is an illegal dog walking state at this time, and there is an illegal state in the monitoring screen at this time. Correspondingly, if there is a person near the traction rope, the key point algorithm can be directly used to extract the key points of the pedestrian, and the pedestrian action can be distinguished based on the binary classification model. When it is determined that the pedestrian posture is a dog walking action, it can be determined to be a normal state. Correspondingly, if the pedestrian posture is not a dog walking action, it can be determined that it is an illegal dog walking state at this time, and there is an illegal state in the monitoring screen at this time.

[0122] Further, Figure 4 FIG. 1 is a flow chart of another data processing method in an embodiment of the present invention. Figure 4 As shown in the figure, because the leash is small in size, it is very likely that the leash is blocked in a single image under the monitoring screen, or a pedestrian passing by happens to intersect with the leash but is not the dog walker, which is easy to be misjudged. Therefore, the embodiment of the present invention can use multiple frames of continuous state tracking of the dog, leash, and dog walker. Figure 4As shown, 10 frames are extracted from the continuous picture frames, the relationship between the three is tracked, and the status of the single frame is obtained respectively. If more than 5 frames are in the normal dog walking state, it is considered that there is no illegal dog walking behavior, otherwise there is illegal dog walking behavior. Bytetrack is used as an embodiment when implementing the tracking algorithm. Of course, other tracking algorithms can also be used, such as FairMOT, TransMOT, CSTracker, etc. Furthermore, the embodiment of the present invention can be applied to the detection of illegal dog walking behavior in monitoring scenarios. A solution based on target detection combined with semantic segmentation and key point action recognition is adopted. Improve the low recognition rate of illegal dog walking behavior caused by the small size of the traction rope target, the variable length of the traction rope, and the special state of having a traction rope but not holding it in the hand.

[0123] Figure 5 FIG. 1 is a schematic diagram of the structure of a data processing device in an embodiment of the present invention. Figure 5 As shown, the device 20 may include:

[0124] A first acquisition module 201 is used to acquire an image to be detected from video data to be detected;

[0125] The first input module 202 is used to input the image to be detected into the target recognition model to recognize the image to be detected and obtain the object information contained in the image to be detected;

[0126] A second input module 203 is used to input the image to be detected into a target classification model to obtain the human action of the human included in the image to be detected when the object information indicates that there is a pet in the image to be detected;

[0127] The behavior determination module 204 is used to determine the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person's action.

[0128] Optionally, the device 20 further includes:

[0129] A second acquisition module is used to acquire multiple sample images and sample labels;

[0130] A third input module, used for inputting the plurality of sample images into an initial detection model to obtain a sample recognition result corresponding to each of the sample images;

[0131] A parameter adjustment module, used to adjust the model parameters of the initial detection model based on the sample recognition result and the sample label, until the training stop condition is reached, and the current initial detection model is determined as the target model;

[0132] The sample labels are used to characterize the object information contained in each of the sample images, and the target model is the target recognition model; or the sample labels are used to characterize the character actions of the characters contained in each of the sample images, and the target model is the target classification model.

[0133] Optionally, when the sample labels are used to characterize the actions of the characters contained in each of the sample images, the third input module includes:

[0134] A third acquisition submodule is used to acquire key points of characters in each of the sample images;

[0135] A vector generation submodule, used to generate a character feature vector based on the character key points;

[0136] The action determination submodule is used to determine the character action based on the character feature vector.

[0137] Optionally, when the sample label is used to characterize object information contained in each of the sample images, the initial detection model at least includes a semantic segmentation model.

[0138] Optionally, the behavior determination module includes:

[0139] A first determination submodule is used for determining that the interaction between the person and the pet in the image to be detected belongs to the preset detection behavior if the distance between the pet position and the traction component position is not greater than a preset distance threshold and if the person action belongs to the preset detection action;

[0140] Or, a second determination submodule is used to determine that the interactive behavior between the person and the pet in the image to be detected belongs to the preset detection behavior when the distance between the pet position and the traction component position is greater than the preset distance threshold.

[0141] Optionally, the first acquisition module is specifically configured to:

[0142] Acquire a plurality of continuous video frames from the video data to be detected as a plurality of images to be detected;

[0143] The device also includes:

[0144] A third determination module, configured to determine the image to be detected as a target image when the interactive behavior belongs to a preset detection behavior;

[0145] The fourth determination module is used to determine that the video data to be detected has the preset detection behavior when the number of the target images existing in the multiple images to be detected is not less than a preset number threshold.

[0146] In summary, the data processing device provided in the embodiment of the present invention obtains an image to be detected from the video data to be detected; inputs the image to be detected into a target recognition model to identify the image to be detected and obtain the object information contained in the image to be detected; when the object information indicates that there is a pet in the image to be detected, inputs the image to be detected into a target classification model to obtain the person action of the person contained in the image to be detected; and determines the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person action. In the embodiment of the present invention, by setting a target recognition model and a target classification model, the interactive behavior between the person and the pet can be detected in combination with object recognition and person action recognition, and by detecting the interactive behavior in combination with person actions, the accuracy of the detection can be improved to a certain extent, and the effect of the interactive behavior detection can be ensured.

[0147] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 9001, a communication interface 9002, a memory 9003 and a communication bus 9004, wherein the processor 9001, the communication interface 9002, and the memory 9003 communicate with each other through the communication bus 9004.

[0148] Memory 9003, used for storing computer programs;

[0149] The processor 9001 is used to implement the data processing method provided in the above embodiment when executing the program stored in the memory 9003.

[0150] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0151] The communication interface is used for communication between the above terminal and other devices.

[0152] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0153] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0154] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0155] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the data processing method described in any one of the above embodiments.

[0156] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0157] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0158] It should be noted that the various data-related processes in the embodiments of the present application are all carried out in compliance with the corresponding data protection laws and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0159] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire an image to be detected from video data to be detected; Inputting the image to be detected into the target recognition model to recognize the image to be detected and obtain the object information contained in the image to be detected; In a case where the object information indicates that a pet exists in the image to be detected, the image to be detected is input into a target classification model to obtain a person action of a person contained in the image to be detected; Based on the object information and / or the person's action, the interactive behavior between the person and the pet in the image to be detected is determined.

2. The method according to claim 1, characterized in that The target model is trained through the following steps: Get multiple sample images and sample labels; Inputting the plurality of sample images into an initial detection model to obtain a sample recognition result corresponding to each of the sample images; Based on the sample recognition result and the sample label, the model parameters of the initial detection model are adjusted until a training stop condition is reached, and the current initial detection model is determined as a target model; The sample labels are used to characterize the object information contained in each of the sample images, and the target model is the target recognition model; or the sample labels are used to characterize the character actions of the characters contained in each of the sample images, and the target model is the target classification model.

3. The method according to claim 2, characterized in that In the case where the sample labels are used to characterize the actions of the characters contained in each of the sample images, obtaining the sample recognition results corresponding to each of the sample images includes: Obtaining key points of characters in each of the sample images; Generate a character feature vector based on the character key points; The character action is determined based on the character feature vector.

4. The method according to claim 2, characterized in that: In the case where the sample labels are used to characterize object information contained in each of the sample images, the initial detection model at least includes a semantic segmentation model.

5. The method according to claim 1, characterized in that The object information includes a person position, a pet position, and a traction component position; and determining the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person action includes: In the case where the distance between the pet position and the traction component position is not greater than a preset distance threshold, if the human action belongs to a preset detection action, determining that the interaction behavior between the human and the pet in the image to be detected belongs to a preset detection behavior; Or, when the distance between the pet position and the traction component position is greater than the preset distance threshold, it is determined that the interactive behavior between the person and the pet in the image to be detected belongs to the preset detection behavior.

6. The method according to any one of claims 1 to 5, characterized in that: The step of acquiring the image to be detected from the video data to be detected comprises: Acquire a plurality of continuous video frames from the video data to be detected as a plurality of images to be detected; The method further comprises: In a case where the interactive behavior belongs to a preset detection behavior, determining the image to be detected as a target image; When the number of the target images existing in the multiple images to be detected is not less than a preset number threshold, it is determined that the preset detection behavior exists in the video data to be detected.

7. A data processing device, characterized in that: The device comprises: A first acquisition module, used for acquiring an image to be detected from video data to be detected; A first input module, used for inputting the image to be detected into the target recognition model to recognize the image to be detected and obtain the object information contained in the image to be detected; A second input module is used for inputting the image to be detected into a target classification model to obtain the person action of the person contained in the image to be detected when the object information indicates that there is a pet in the image to be detected; The behavior determination module is used to determine the interactive behavior between the person and the pet in the image to be detected based on the object information and / or the person's action.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program, characterized in that When the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images