Target behavior detection method and device, electronic equipment and medium

Abstract

The embodiment of the invention discloses a target behavior detection method and device, electronic equipment and a medium. The method comprises the following steps: for a sample image containing a target, adjusting features of a preset area in the sample image to obtain a new sample image, and taking the new sample image and an initial sample image as the sample image together to perform quantity expansion on the sample image; training the initial model in the process of carrying out key point prediction on the sample images after quantity expansion through the initial model and in the process of carrying out target attitude classification according to the predicted key points; and when the loss function meets the requirement, training is ended to obtain a target posture recognition model, the posture of the target in the target image is recognized based on the target posture recognition model, and the behavior of the target is determined according to the posture of the target. According to the scheme, the number expansion and the quality improvement of the sample images are realized, the accuracy of model identification is further improved, and the identification of various complex environments can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of behavior recognition, and particularly to a target behavior detection method and device, electronic equipment and a medium. BACKGROUND

[0002] Behavior recognition is a classification task, and the input is a cropped video sequence, and the output is the behavior of the target in the video sequence. The behavior recognition technology is usually deployed in monitoring cameras, video review and other application scenarios, and usually uses a related trained model to detect the behavior of the target. However, in the training process of these models, due to the problems such as a small amount of sample data or poor quality of the obtained sample data, the recognition efficiency of the related model is low, and the model cannot well recognize a variety of complex environments. SUMMARY

[0003] Embodiments of the present application provide a target behavior detection method and device, electronic equipment and a medium, which realize the expansion of the number and the improvement of the quality of sample images, thereby improving the accuracy of model recognition and enabling the recognition of a variety of complex environments.

[0004] According to an aspect of the present application, a target behavior detection method is provided, which comprises:

[0005] For a sample image containing a target, the features of a preset region in the sample image are adjusted to obtain a new sample image, and the new sample image and the initial sample image are collectively used as sample images to expand the number of the sample images.

[0006] Wherein, the adjustment of the features of the preset region comprises adding the features of the target in the preset region; the sample image is labeled with key points and target poses;

[0007] The initial model is trained in the process of key point prediction of the sample image with expanded number by the initial model and in the process of target pose classification according to the predicted key points;

[0008] The training is ended to obtain a target pose recognition model when the loss function meets the requirements, so as to recognize the pose of the target in the target image based on the target pose recognition model, and determine the behavior of the target according to the pose of the target.

[0009] According to an aspect of the present application, a target behavior detection device is provided, which comprises:

[0010] A sample expansion module is configured to extract and recombine a target region of interest in a sample image containing a target to expand the number of the sample image; wherein the sample image is labeled with key points and target poses.

[0011] a training module configured to train the initial model in a process of predicting key points of the sample images after quantity expansion by the initial model and in a process of classifying poses of the target according to the predicted key points;

[0012] a target pose recognition model determination module configured to end the training to obtain a target pose recognition model in a case where the loss function meets a requirement, to recognize a pose of a target in a target image based on the target pose recognition model, and to determine a behavior of the target according to the pose of the target.

[0013] According to another aspect of the present application, an electronic device is provided, which comprises:

[0014] at least one processor; and

[0015] a memory in data processing connection with the at least one processor; wherein,

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the target behavior detection method of any of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the target behavior detection method of any of the embodiments of the present application when executed by the processor.

[0018] The technical solution of the embodiments of the present application is that, for a sample image containing a target, the features in a preset region of the sample image are adjusted to obtain a new sample image, the new sample image and the initial sample image are jointly used as sample images to expand the quantity of the sample images; the initial model is trained in a process of predicting key points of the sample images after quantity expansion by the initial model and in a process of classifying poses of the target according to the predicted key points; and the training is ended to obtain a target pose recognition model in a case where the loss function meets a requirement, to recognize a pose of a target in a target image based on the target pose recognition model, and to determine a behavior of the target according to the pose of the target. The above solution adjusts the features in the preset region of the sample image, including adding features of the target in the preset region, to obtain a new sample image, realizes quantity expansion and quality improvement of the sample image, and further improves the accuracy of model recognition, and can realize recognition of various complex environments.

[0019] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent through the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a target behavior detection method provided in an embodiment of this application;

[0022] Figure 2 A schematic diagram of a model structure provided for an embodiment of this application;

[0023] Figure 3 A schematic diagram of an improved structure of a self-attention layer provided in an embodiment of this application;

[0024] Figure 4 A flowchart of another target behavior detection method provided in the embodiments of this application;

[0025] Figure 5a A schematic diagram of key points of a dog provided in an embodiment of this application;

[0026] Figure 5b A schematic diagram of a target region of interest provided in an embodiment of this application;

[0027] Figure 5c A schematic diagram of a global region of interest provided in an embodiment of this application;

[0028] Figure 5d A schematic diagram of a sample image after adding target features to a preset area, as provided in an embodiment of this application;

[0029] Figure 6 A flowchart illustrating yet another target behavior detection method provided in this application embodiment;

[0030] Figure 7a This is a schematic diagram illustrating an overexposed target in a sample image provided in an embodiment of this application.

[0031] Figure 7b This application provides a schematic diagram of a sample image after transforming the color space parameters of a preset area, as an embodiment of the present application.

[0032] Figure 8a A schematic diagram of key points of a person provided in an embodiment of this application;

[0033] Figure 8bA schematic diagram of target key points of a person provided by an embodiment of the present application;

[0034] Figure 9 A schematic diagram of a triangle formed by hip-knee-ankle three skeletal point coordinates of a person provided by an embodiment of the present application;

[0035] Figure 10 A structural schematic diagram of a target behavior detection device provided by an embodiment of the present application;

[0036] Figure 11 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable personnel in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0038] It should be understood that each step described in the method embodiments of the present application can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0039] It should be noted that the terms "first", "second", "third", "fourth", "actual", "preset" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0040] Figure 1 A flowchart of a target behavior detection method provided by an embodiment of the present application, which can be applicable to the case of identifying and detecting the behavior of a target in an image. The method can be executed by a target behavior detection device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in FIG. 1, the target behavior detection method provided by an embodiment of the present application includes the following steps.Figure 1 As shown, the target behavior detection method in this embodiment may include the following steps:

[0041] S110. For a sample image containing a target, adjust the features of a preset region in the sample image to obtain a new sample image. Use the new sample image and the initial sample image together as a sample image to expand the number of sample images. The adjustment of the features of the preset region includes adding the features of the target in the preset region. The sample image is labeled with key points and the target pose.

[0042] The target can be a pre-defined target, i.e., a pre-specified target that needs to be detected for behavior recognition. The target can be a human or an animal in an image. Sample images can be images from the actual scene acquired based on different dimensions such as the acquisition scene, angle, and time period, or relevant images configured based on the actual target behavior detection requirements. For example, video streams can be acquired from surveillance cameras or hard disk recorders, and sample images can be determined based on the video stream. Preferably, images with clear target outlines and meeting pixel requirements can be used as sample images.

[0043] A preset region refers to part or all of the target area in a sample image. Behavior detection of the target can be performed based on the content of the image within the preset region. Specifically, the sample image is marked with key points and target pose. Key points are location points that can reflect the contour or behavioral information of the target in the sample image, and the target pose is the result of the target's behavior in the pre-labeled sample image, i.e., the target pose.

[0044] In practice, there may be issues such as a limited number of available sample images or samples containing little useful feature information. This results in a small number of sample images suitable for model training, or images of poor quality, which negatively impacts training accuracy. To address this, new sample images can be obtained by adjusting features in predetermined regions of the sample images. These new sample images, along with the initial sample images, can then be used together to expand the quantity and improve the quality of the sample images, thereby enhancing training accuracy to some extent.

[0045] Specifically, adjusting the features of the preset area includes adding the features of the target within the preset area.

[0046] The acquired sample images may lack sufficient feature information related to the target behavior. To address this, feature enhancement can be achieved by adding target features to a predetermined region of the sample image. This enriches the feature information within the predetermined region, resulting in a new sample image with richer feature information, which is beneficial for improving the accuracy of model training. For example, a new sample image can be obtained by cropping the image corresponding to a predetermined region from the sample image and merging the cropped image into the initial sample image.

[0047] S120. The initial model is trained during the process of predicting key points in the expanded sample images using the initial model and during the process of classifying target poses based on the predicted key points.

[0048] The initial model can be used to predict keypoints in sample images, or it can be used to classify target poses based on the predicted keypoints. The target pose is predefined, and the target pose classification process can be performed using a classifier within the model. For example, a YOLOv5 model can be used as the initial model, and augmented sample images can be used to train the initial model for relevant processes.

[0049] For example, the initial model framework is divided into three parts. The first part is the backbone network, in which a ResNet architecture can be used, retaining only the first four convolutional layers to extract high-level features of the image. The second part is the Transformer architecture, which includes an encoder containing T self-attention layers and a feedforward neural network in sequence, which can compress the feature maps extracted by the backbone into keypoint location representations. The third part is the Head layer, which can be used to predict keypoints and classify the target pose based on the predicted keypoints. Figure 2 As shown, Figure 2 This is a schematic diagram of a model structure provided for an embodiment of this application.

[0050] As an optional but not limited implementation, the process of predicting keypoints in the expanded sample images using an initial model can include:

[0051] In the process of processing sample image features based on the self-attention mechanism, the maximum number of elements in each row of the initial similarity matrix are normalized, and the other elements are set to zero to obtain the final similarity matrix.

[0052] This application improves the self-attention layer in the Transformer. The self-attention mechanism is implemented by calculating a similarity matrix, which reflects the correlation between different elements in the input sequence. In the process of extracting sample image features based on the self-attention mechanism, an initial similarity matrix can be calculated first. Then, a predetermined number of the largest element values ​​in each row of the initial similarity matrix are normalized, and other element values ​​are set to zero. The processed matrix is ​​then used as the final similarity matrix. By normalizing the predetermined number of largest element values ​​in the initial similarity matrix and setting other element values ​​to zero, the ability to distinguish between the detected target and the background is improved, enhancing the anti-interference capability of the target features, making the output features less susceptible to noise and background influences, and improving the detection accuracy of key points.

[0053] Specifically, Figure 3 This is a schematic diagram of an improved structure of a self-attention layer provided in an embodiment of this application. Figure 3 As shown, the initial similarity matrix is ​​first obtained by multiplying the Query (query vector matrix) and Key (key vector matrix) and then performing scaling and masking operations. Next, the C largest elements are selected from each row of the initial similarity matrix, and the Softmax function is used to normalize the C element values ​​in each row. Finally, the Scatter function is used to pad the remaining elements with zeros, resulting in the final similarity matrix. Multiplying the final similarity matrix by the Value (value vector matrix) yields the dynamic attention weight matrix.

[0054] The last self-attention layer in the Transformer (also an improvement on the above scheme) acts as an aggregator. It summarizes the contributions of all image locations based on the attention scores of each extracted feature vector and predicts the most influential keypoints at those locations. Specifically, it identifies the most representative or influential locations by calculating the attention score of each potential keypoint, i.e., calculating the maximum activation value. Where i * Let S represent the true location of a keypoint, and h represent the given image. i* This represents the attention score. The Head layer selects the keypoint with the greatest influence in the predicted location based on the maximum activation value, and uses it as the predicted keypoint. Then, all predicted keypoints are finally passed to the classifier to classify the target pose.

[0055] S130. If the loss function meets the requirements, the training ends and the target pose recognition model is obtained. The target pose recognition model is used to identify the pose of the target in the target image and determine the target's behavior based on the target's pose.

[0056] The loss function reflects the difference between the model's predictions and the actual results. The loss function can be set according to specific needs. Training can end when the loss function meets the requirements, resulting in the target pose recognition model. This model can be used to identify the pose of a target in an image and determine the target's behavior based on that pose.

[0057] The technical solution of this application embodiment, for a sample image containing a target, adjusts the features of a preset region in the sample image to obtain a new sample image. The new sample image and the initial sample image are used together as a sample image to expand the number of sample images. The initial model is trained during the process of predicting key points in the expanded sample image using the initial model, and during the process of classifying the target pose based on the predicted key points. Training ends when the loss function meets the requirements, resulting in a target pose recognition model. This model is then used to identify the pose of the target in the target image and determine the target's behavior based on the pose. This solution, by adjusting the features in a preset region of the sample image, including adding target features to the preset region, to obtain a new sample image, achieves both quantity expansion and quality improvement of the sample images, thereby improving the accuracy of model recognition and enabling the recognition of various complex environments.

[0058] Figure 4 This is a flowchart of another target behavior detection method provided in this application embodiment. The technical solution of this embodiment further optimizes the process of adding target features to a preset area based on the technical solutions of the above embodiments. Solutions not described in detail in this embodiment can be found in the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 4 As shown, the target behavior detection method in this embodiment may include the following steps:

[0059] S410. For a sample image containing a target, determine the target key points from the key points marked in the sample image, and determine the target region of interest based on the region corresponding to the target key points; wherein the number of target key points is less than or equal to the total number of key points.

[0060] Targets in sample images are usually labeled with multiple key points. However, when a target performs a specific type of behavior, some of these key points become more representative. Therefore, when performing behavior detection on a target, the behavior detection can be completed quickly based on these representative key points.

[0061] Specifically, target key points are key points that can reflect the key behaviors of the target, and the number of target key points is less than or equal to the total number of key points.

[0062] For example, taking a dog in an image as the target, key points of the dog could be its eyes, nose, ears, tail, front paws, and hind paws. To identify and detect a dog's defecation behavior, since the dog's posture features during defecation include an arched spine, a raised tail, and bent hind legs, the key points corresponding to these posture features can be labeled and used as target key points. Specifically, the following 10 points can be used as target key points: the dog's neck, back, tail, tail tip, left kneecap, right kneecap, left ankle bone, right ankle bone, left paw, and right paw. Figure 5a This is a schematic diagram of key points of a dog provided in an embodiment of this application. For example... Figure 5a As shown in the figure, the red dots are the target key points marked for dogs.

[0063] As an optional but not limited implementation, determining the region of interest based on the region corresponding to the target key points may include the following steps A1-A2:

[0064] Step A1: For each target key point, determine the preset number of related key points that are closest to the target key point.

[0065] Specifically, a Cartesian coordinate system can be established with the lower left corner of the sample image as the origin to determine the coordinates of each labeled keypoint in the sample image. For each identified target keypoint, the Euclidean distance from the target keypoint to any other keypoint is calculated and sorted by minimum distance. Then, a preset number of keypoints with the smallest distance to the target keypoint are determined as relevant keypoints. The preset number can be 3 or 4, depending on the actual needs.

[0066] Step A2: Select the smallest bounding polygon containing the target key points and related key points as the target region of interest.

[0067] After identifying the target key points and a preset number of related key points, the minimum bounding polygon of these key points can be used as the target region of interest. The minimum bounding polygon encompasses both the target key points and related key points, and the corresponding area of ​​the image is the target region of interest. Specifically, the target region of interest is the image area containing key target behaviors, and it concentrates key image content that enables behavior detection of the target.

[0068] It should be noted that, in the specific implementation process, based on the target key points and related key points, in addition to determining the smallest bounding polygon, it can also be a bounding polygon with a larger range. The specific choice can be made flexibly according to actual needs.

[0069] For example, taking a target keypoint with one objective and three related keypoints, these four keypoints can be connected to form an irregular quadrilateral. The minimum bounding rectangle containing these four keypoints can then be taken as the target region of interest. Specifically, the minimum and maximum values ​​of the x and y coordinates of the four vertices of the quadrilateral are calculated to obtain (minx... i ,miny i ,maxx i ,maxy i Among them, for minx i and maxx i Draw perpendicular lines to the X-axis respectively; for miny i and maxy i Draw perpendicular lines to the Y-axis from each of the two axes; this will result in two intersection points (x, y). i y i ), (x j y j These two intersection points are the two opposite vertices of the minimum bounding rectangle. Based on the coordinates of these two intersection points, the corresponding minimum bounding rectangle can be determined, and the screen area corresponding to the minimum bounding rectangle at this time is taken as the target region of interest. Figure 5b A schematic diagram of a target region of interest provided for an embodiment of this application, such as... Figure 5b As shown in the figure, the area enclosed in the diagram is the target region of interest.

[0070] It should be noted that although different target key points can be selected, the bounding rectangles determined for different target key points and the corresponding related key points may be exactly the same. In this case, only one bounding rectangle can be kept to enrich the feature information of the image in subsequent schemes, while other identical bounding rectangles can be deleted.

[0071] S420. Extract the target region of interest from the sample image and add the target region of interest to the preset area to expand the number of sample images.

[0072] After identifying the target region of interest (ROI), it can be extracted from the sample image, for example, through image cropping. The extracted ROI is then added to a preset region within the initial sample image, and the resulting image is used as a new sample image. Subsequently, both the new and initial sample images can be used together to expand the sample image set.

[0073] As an optional but not limited implementation, adding a target region of interest to a preset area may include the following steps B1-B2:

[0074] Step B1: Use the smallest bounding polygon of all key points in the sample image as the global region of interest.

[0075] For all the key points annotated in the sample image, the smallest bounding polygon of these key points can be taken as the global region of interest. Since the smallest bounding polygon includes all the key points of the target in the image, the global region of interest corresponding to the smallest bounding polygon is equivalent to including most of the feature information of the target.

[0076] It should be noted that in the specific implementation process, based on all key points in the sample image, either the minimum bounding polygon or the bounding polygon can be determined, and the specific choice can be made flexibly according to actual needs.

[0077] For example, Figure 5c A schematic diagram of a global region of interest provided for an embodiment of this application, such as... Figure 5c As shown in the figure, the area enclosed in the diagram is the global region of interest, which contains all the key points of the dog.

[0078] Step B2: Use the region outside the global region of interest in the sample image as the preset region, and add the target region of interest to the preset region.

[0079] Since the global region of interest already includes most of the target's feature information, while the regions outside the global region of interest in the sample image may not have much useful information, relevant feature information of the target can be added to these regions that do not have much useful information, so that the entire sample image has richer feature information.

[0080] Specifically, after determining the global region of interest, the region outside the global region of interest in the sample image is taken as the preset region, and the previously obtained target region of interest is added to the preset region. This makes the preset region also contain the feature information of the target, which to a certain extent increases the richness of the data content in the sample image and is more conducive to improving the accuracy of model training.

[0081] Specifically, after obtaining the target region of interest (ROI) through image cropping, the ROI can be randomly stitched to one of the four preferred positions: the top left, bottom left, top right, and bottom right corners of the initial sample image. Alternatively, it can be stitched to other positions such as the center left or right. During the stitching process, the target stitching position can be determined based on the first intersection-union ratio (IU / U) between the ROI and the global region of interest (GRI) in the sample image. The IU / U is the ratio of the areas of the ROI and the GRI. If the calculated IU / U is 0, it means that the ROI and the GRI have no overlap, and the ROI can be directly stitched to the current position. If the calculated IU / U is not 0, it means that the ROI and the GRI have overlap, and the ROI can be further stitched to the remaining positions, and a new IU / U is calculated. This process continues until a position with an IU / U is found that is 0 before stitching is performed.

[0082] Optionally, if the first intersection-union ratio (IU) calculated after traversing all positions is not zero, it is preferable to stitch the target region of interest (ROI) to the position corresponding to the minimum IU, and simultaneously scale the ROI proportionally according to a preset scaling factor until the first IU between the scaled ROI and the global region of interest is zero. For example, the ROI can be scaled proportionally by a scaling factor of 5%.

[0083] For example, Figure 5d This application provides an embodiment of a sample image after adding target features to a preset area, as shown in the following diagram. Figure 5d As shown. Areas outside the global region of interest can be used as preset regions. This is achieved by adding the target region of interest to the preset region, as shown in the example. Figure 5d Adding it to the right of the global region of interest will enrich and expand the feature information in the initial sample image.

[0084] As an optional but not limited implementation, adding the target region of interest after the preset region further includes: if the target region of interest exceeds the sample image, adjusting the target region of interest so that the target region of interest is within the sample image; wherein the adjustment method includes at least one of scaling, translation, and rotation.

[0085] Specifically, after extracting the target region of interest (ROI), the obtained ROI may be too large. If an excessively large ROI is directly added to the preset area, it is very easy for the size of the new sample image to exceed the size of the initial sample image. The size inconsistency between multiple sample images will directly affect the accuracy of model training. Therefore, it is necessary to adjust the extracted ROI so that the ROI remains within the sample image after stitching and does not exceed the size range of the initial sample image. The adjustment method includes at least one of scaling, translation, and rotation. The specific method can be determined according to actual needs.

[0086] As an optional but not limited implementation, an image without a target can be used as a background image, the target region of interest obtained in the above steps can be directly stitched onto the background image, and new sample images can be obtained by randomly scaling, flipping, or other methods.

[0087] S430. The initial model is trained during the process of predicting key points in the expanded sample images using the initial model and during the process of classifying target poses based on the predicted key points.

[0088] S440. If the loss function meets the requirements, the training ends and the target pose recognition model is obtained. The target pose is recognized based on the target pose recognition model, and the target behavior is determined based on the target pose.

[0089] The technical solution of this application embodiment, for a sample image containing a target, adjusts the features of a preset region in the sample image to obtain a new sample image. The new sample image and the initial sample image are used together as a sample image to expand the number of sample images. The initial model is trained during the process of predicting key points in the expanded sample image using the initial model, and during the process of classifying the target pose based on the predicted key points. Training ends when the loss function meets the requirements, resulting in a target pose recognition model. This model is then used to identify the pose of the target in the target image and determine the target's behavior based on the pose. This solution, by adding target features to a preset region of the sample image, enriches the feature information in the sample image. The resulting new sample image expands the number of sample images and improves their quality, thereby enhancing the accuracy of the model's recognition and enabling the recognition of various complex environments.

[0090] Figure 6This is a flowchart illustrating another target behavior detection method provided in this application embodiment. This embodiment further optimizes the process of adjusting the features of the preset region in the aforementioned embodiments. This application embodiment enriches the number of sample images by adjusting the pixel values ​​of each pixel. Solutions not described in detail in this embodiment are found in the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 6 As shown, the target behavior detection method in this embodiment may include the following steps:

[0091] S610. For a sample image containing a target, convert each pixel in a preset region from the RGB color space to the HSV color space. Specifically, for a sample containing a target, each pixel in a preset region can be converted from the RGB color space to the HSV color space. The preset region refers to part or all of the image region containing the target in the sample image. For example, the R, G, B information of each pixel in the sample image can be converted into H, S, V information using the following formula:

[0092] V = max(R, G, B);

[0093]

[0094]

[0095] Where (R, G, B) represent the red, green, and blue color components of a pixel, and (H, S, V) represent the chroma, saturation, and lightness attributes of a pixel. Based on the judgment conditions and specific formula content above, the conversion of pixels from the RGB color space to the HSV color space can be achieved.

[0096] S620. Multiply the color space parameters in the HSV color space by the preset scaling factor to obtain the adjusted parameters in the HSV color space.

[0097] The magnification factor can range from [0.5, 1.5], and the preset magnification factor can be randomly selected based on actual needs. The adjusted parameters include the brightness, chroma, and saturation of each pixel. Specifically, after converting the pixels into their corresponding HSV color space information, a magnification factor can be randomly multiplied by the specific H, S, and V values ​​to adjust the brightness, chroma, and saturation of each pixel in the sample image, thereby enhancing the image data and improving the quality of the sample image.

[0098] Optionally, the process of multiplying each pixel by the magnification factor can be repeated a preset number of times to obtain a preset number of new sample images equal to the number of iterations. For example, repeating the process of multiplying by the magnification factor 3 times will generate 3 different new sample images.

[0099] Multiplying by a magnification factor can improve the display of individual pixels in the initial sample image, but it may also directly lead to overexposure or underexposure in certain areas. If this occurs in areas corresponding to critical actions of the target, it will hinder subsequent feature recognition using the sample image. For example, Figure 7a This is a schematic diagram illustrating an overexposed target in a sample image provided in an embodiment of this application, such as... Figure 7a As shown, the chest and forelegs of the dog in the picture are too bright, causing the image details to be lost.

[0100] As an optional but not limited implementation, after multiplying the color space parameters in the HSV color space by a preset scaling factor to obtain the adjusted parameters, the following steps C1-C5 may also be included:

[0101] Step C1: Determine the average brightness of the region in the preset area of ​​the sample image, and the average brightness of the image in the sample image.

[0102] After multiplying the color space parameters by a preset scaling factor to obtain adjusted parameters, there may be overexposure or underexposure. Although such sample images record relevant target behaviors, overexposure or underexposure is not conducive to subsequent model training, hinders the model from recognizing key information, and easily leads to the loss of key information. Efforts can be focused on adjusting and improving overexposure or underexposure in preset areas of the sample images.

[0103] The average brightness can be determined based on the brightness values ​​of all pixels within the region. Specifically, it can be determined by summing the brightness values ​​of all pixels and then calculating the average. For example, the average brightness of a sample image can be calculated using the following formula:

[0104]

[0105] Among them, V meanall The value represents the average brightness of the sample image, where n represents the number of pixels in the sample image region, and Vi represents the specific brightness value corresponding to each pixel.

[0106] Similarly, the average brightness of the preset area of ​​the sample image can also be calculated based on the principle of the above formula. The difference is that the number of pixels used in the calculation is the number of pixels contained in the preset area of ​​the sample image.

[0107] Step C2: If the average brightness of the region is within the first preset range and the average brightness of the region is greater than the average brightness of the image, or if the average brightness of the region is within the second preset range and the average brightness of the region is less than the average brightness of the image, then for each pixel in the preset region, the first exponent of the adjusted brightness of that pixel is taken as the final brightness of that pixel.

[0108] The first index is determined based on the ratio of the average brightness of the region to the average brightness of the image. Final brightness refers to the adjusted brightness value of each pixel. The first and second preset intervals can be divided based on actual needs. The first preset interval is larger than the second preset interval; a higher interval value indicates a brighter, more overexposed image, while a lower interval value indicates a darker, less bright image.

[0109] If the average brightness of a region falls within the first preset range and is greater than the average brightness of the image, it indicates severe overexposure within the preset region. If the average brightness of a region falls within the second preset range and is less than the average brightness of the image, it indicates severe underexposure within the preset region. To address these severe overexposure or underexposure issues within the preset region, the brightness of each pixel within that region can be significantly adjusted. Specifically, for each pixel in the preset region, the final brightness of that pixel can be calculated by raising its adjusted brightness to the first power, thus achieving a substantial adjustment to the brightness of each pixel within the preset region.

[0110] Step C3: If the average brightness of the area is within the third preset range, then the brightness of the pixels in the preset area will not be adjusted.

[0111] The third preset interval is located between the first and second preset intervals; that is, the first preset interval is greater than the third preset interval, and the third preset interval is greater than the second preset interval. For example, the first preset interval can be set to (0.75, 1], the third preset interval can be set to [0.25, 0.75], and the second preset interval can be set to [0, 0.25].

[0112] Specifically, if the average brightness of the region is within the third preset range, it means that the brightness of each pixel in the sample image meets the requirements, and the brightness of the pixels in the preset region does not need to be adjusted.

[0113] Step C4: If the average brightness of the region is within the first preset interval and the average brightness of the region is less than or equal to the average brightness of the image, or if the average brightness of the region is within the second preset interval and the average brightness of the region is greater than or equal to the average brightness of the image, then for each pixel in the preset region, the second exponent of the adjusted brightness of that pixel is taken as the final brightness of that pixel.

[0114] The second index is determined based on the ratio of the average image brightness to the average regional brightness. If the average regional brightness is within the first preset range and is less than or equal to the average image brightness, it indicates that while the image within the preset region is overexposed, the overexposure is not particularly severe. If the average regional brightness is within the second preset range and is greater than or equal to the average image brightness, it indicates that while the image within the preset region is underexposed, the underexposedness is not particularly severe. For cases where overexposure or underexposedness is not particularly severe, the brightness of each pixel within the preset region can be adjusted slightly. Specifically, for each pixel within the preset region, the second exponent of the adjusted brightness can be used as the final brightness of that pixel, thus achieving a slight adjustment to the brightness of each pixel within the preset region.

[0115] It should be noted that the first and second indices essentially reflect the magnitude of adjustment to pixel brightness information; the larger the index, the greater the adjustment magnitude; the smaller the index, the smaller the adjustment magnitude.

[0116] For example, the adjusted luminance component of each pixel can be calculated based on the following formula:

[0117]

[0118] Among them, V * V represents the adjusted luminance component for each pixel. meanall V represents the average brightness of the sample images. mean-2 Let α be the average brightness of the preset region in the sample image, and α be a scaling factor, typically √2. In practice, the brightness components after pixel adjustment can be calculated based on the interval conditions satisfied by the specific brightness values ​​of each pixel and according to the corresponding specific function.

[0119] Step C5: Calculate the final chromaticity and final saturation of each pixel in the preset area based on the final brightness.

[0120] Here, final chroma refers to the adjusted chroma value of a pixel, and final saturation refers to the adjusted saturation value of a pixel. After adjusting the brightness values ​​of each pixel in the preset area, the final chroma and final saturation of each pixel can be calculated. Specifically, this can be done synchronously based on the existing relationships between the brightness, chroma, and saturation of each pixel.

[0121] S630 converts each pixel in the preset area from the HSV color space back to the RGB color space.

[0122] By adjusting the H, S, and V components of each pixel, it is equivalent to performing data enhancement on the initial sample image, overcoming the problems of overexposure or underexposure to a certain extent. Alternatively, the pixels in a preset area can be converted from the HSV color space back to the RGB color space to form a new sample image, which is then used together with the original sample image as a sample image, thereby expanding the number of sample images. Figure 7b This application provides a schematic diagram of a sample image after transforming the color space parameters of a preset area, as shown in the embodiment of the present application. Figure 7b As shown, by adjusting the color space parameters of the preset area, the overexposure or underexposure in the sample image was improved to a certain extent, thus enhancing the image quality.

[0123] S640. The initial model is trained during the process of predicting key points in the expanded sample images using the initial model and during the process of classifying target poses based on the predicted key points.

[0124] S650. If the loss function meets the requirements, the training ends and the target pose recognition model is obtained. The target pose recognition model is used to identify the pose of the target in the target image and determine the target's behavior based on the target's pose.

[0125] As an optional but not limited implementation, the process of determining the loss function in keypoint prediction of augmented sample images using an initial model may include the following steps D1-D3:

[0126] Step D1: Determine the first difference data between the labeled keypoints in the sample image and the predicted keypoints of the initial model.

[0127] During model training, there may be instances where key points of targets in sample images are not identified or are lost. For example, Figure 5a Keypoints at the tip of a dog's tail may not be recognized, and keypoints at the very edges may be invisible. To address this, a loss function can be set to improve the model's prediction accuracy for these invisible keypoints.

[0128] Specifically, the first difference data can be determined based on the labeled keypoints in the sample image and the predicted keypoints of the initial model. Here, the predicted keypoints correspond to the labeled keypoints, and the first difference data reflects the degree of deviation between the labeled keypoints in the sample image and the predicted keypoints of the initial model.

[0129] Specifically, you can first label the actual key points (x i y i The encoding mapping yields a corresponding heatmap, which is a two-dimensional array where each element represents the probability that a point in the image is a keypoint. A higher value indicates a more likely location is a keypoint. The value of a specific point (x, y) on the heatmap... Where σ is a fixed value, typically 1 / 32 of the heatmap width. Then, the mean squared error (MSE) between the actual coordinates of the keypoints on the heatmap and the predicted coordinates is calculated using the following formula:

[0130]

[0131] Among them, L MSE Let K be the mean squared error function, where K represents the number of key points, m is the width of the heatmap, and n is the height of the heatmap. The values ​​in the heatmap are based on the actual mapping of key points. To predict the values ​​in the heatmap of keypoint mapping.

[0132] Step D2: Determine the second difference data between the labeled keypoints in the sample image and the predicted labeled keypoints that have the greatest impact on those keypoints.

[0133] The second set of difference data reflects the correlation between the labeled keypoints in the sample image and the predicted keypoints that have the greatest influence on those keypoints. This correlation can be reflected by the distance between the two keypoints, which is essentially a distance constraint. The prediction process for the keypoint that has the greatest influence on a given keypoint can be implemented based on the already obtained labeled keypoints and the weights calculated by the self-attention layer. Specifically, during model training, based on the labeled keypoints in the sample image, for a given keypoint 'a', the self-attention layer can calculate the attention weight values ​​between other keypoints and keypoint 'a'. The keypoint with the largest attention weight value is the one that has the greatest influence on keypoint 'a'. Based on this relationship, the model can predict the keypoint that has the greatest influence on keypoint 'a'.

[0134] For example, such as Figure 5aAs shown, there is a certain distance between the key point at the end of the dog's tail and the key point at the head of the tail. By using the distance between the two as a distance constraint, during the model training process, the model will not only identify the key point of the dog's tail, but also learn that the key point of the dog's tail has the greatest influence on the key point at the end of the dog's tail, and the distance between the two is related.

[0135] By introducing keypoint distance constraints, the model can better learn the spatial relationships between keypoints. After continuous learning based on this content, even when some keypoints are not visible, the model can use the relationships between other visible keypoints to infer the location of the invisible keypoints and achieve the prediction of the invisible keypoints.

[0136] Specifically, the actual labeled keypoints f can be calculated. i (x i y i The points marked with the greatest impact on the prediction of this key point. The mean distance between them is used as the second difference data. For example, the second difference data can be represented by a distance constraint function, as shown in the following formula:

[0137]

[0138] Among them, L DC This is the distance constraint function; The constraint coefficient is typically set to 0.3; S represents the area of ​​the heatmap corresponding to the key point that has the greatest impact on the prediction of a certain key point.

[0139] Step D3: Determine the loss function based on the first difference data and the second difference data.

[0140] Since the first difference data reflects the difference between the actual labeled keypoints and the model-predicted keypoints, and the second difference data reflects the correlation between invisible keypoints and the visible keypoints that have the greatest impact on the invisible keypoints, the loss function can be determined based on the first and second difference data.

[0141] Specifically, the loss function can be set as the sum of the mean squared error function and the distance constraint function, and the specific loss function can be expressed by the following formula:

[0142]

[0143] Then, the loss function can be determined based on the loss function. The solution of this application can relocate some occluded key points based on the loss function, realize the full extraction of effective feature information in the image, and thus improve the accuracy of the model in key point detection.

[0144] As an optional but not limited implementation, after training to obtain the target pose recognition model while the loss function meets the requirements, the target pose recognition model can be applied to the detection and judgment of whether dog feces need to be cleaned up in real-world scenarios. Specifically, this can include the following steps E1-E4:

[0145] Step E1: Input the animal image in the target image into the target pose recognition model to determine the animal pose.

[0146] The target image can be an actual image acquired for detection and recognition. For example, the target image can be acquired based on a video stream captured by a surveillance camera.

[0147] As an optional but not limited implementation, the process of determining the target image includes the following steps F1-F2:

[0148] Step F1: Perform target detection and tracking on the animals in the acquired images, and calculate the overlap data of the detection boxes of the same animal in adjacent acquired images;

[0149] The captured images are the actual images obtained. For example, they could be images acquired through a surveillance camera. However, some of the captured images may not contain the relevant footage for animal behavior detection; behavior detection will only be performed if the relevant footage of the animal is present.

[0150] Taking the detection of defecation behavior in a dog in an image as an example, when a dog is defecating, its body maintains a certain posture, and it resumes movement after defecation. Therefore, when performing target detection and tracking on animals in captured images, the overlap data of the detection boxes of the same animal in adjacent frames can be used to reflect whether the animal is continuously performing a certain action. The overlap data can be represented as the intersection-over-union ratio (IoU) of the detection boxes of the same animal in adjacent frames.

[0151] Specifically, taking the image of a dog in the target image as an example, the first step is to track the animal in the target image, and then determine the animal image in the target image based on the tracked animal. Specifically, a target tracking algorithm can be used to track the animal in the target image. After successfully tracking the dog in the target image, target tracking continues for dog detection, calculating the intersection-union ratio (IU) of the detection boxes for the dog in the preceding and following video frames. Specifically, the IU can be calculated based on the specific coordinates of the detection boxes for the dog in the preceding and following frames.

[0152] Step F2: If the overlap data is greater than the preset overlap data, then the images acquired from adjacent frames are used as the target images.

[0153] In this context, adjacent frame images refer to adjacent frames within the captured images. If the overlap data is greater than the preset overlap data, it indicates that the same animal in the adjacent frame images has maintained a certain posture in the same position for a long time, and the animal's specific behavioral state has not ended. Therefore, these adjacent frame images contain relevant important scene content, and these adjacent frame images can be preserved and subsequently used as target images for behavior detection.

[0154] If the overlap data is less than the preset overlap data, it means that the specific behavior of the same animal in the adjacent frame images has ended and the animal's position has changed. Therefore, these adjacent frame images do not contain relevant important scene content and do not need to be retained or behavior detected.

[0155] After obtaining the animal image from the target image, the animal image can be input into the target pose recognition model to determine the animal pose.

[0156] Step E2: If the animal's posture indicates that it is defecating, then input the image of the person closest to the animal in the target image into the target posture recognition model to determine the person's posture.

[0157] As an optional but not limited implementation, the process of determining whether an animal is exhibiting defecation behavior based on its posture includes:

[0158] If the ratio of the number of target images whose animal postures match the preset animal postures to the total number of target images reaches the preset ratio, then it is determined that the animal is exhibiting excretion behavior.

[0159] Specifically, if an animal pose in a target image is detected to match a preset animal pose, the frame number of that target image is recorded, and the process continues to detect the remaining target images, recording the frame numbers of those that meet the requirements. Finally, the number of target images with recorded frame numbers is counted. If the ratio of the number of recorded target images to the total number of target images reaches a preset ratio, it is determined that the animal has exhibited defecation behavior. For example, counting can be done using a queue or a linked list; details are not specified here.

[0160] Optionally, if the total number of target images is large, frame skipping detection can be performed during the process of checking whether the animal pose in each target image conforms to the preset animal pose, instead of performing frame-by-frame detection. The number of frame skipping intervals can be determined based on the total number of target images. If the total number of target images is large, the number of frame skipping intervals can be set larger; if the total number of target images is small, the number of frame skipping intervals can be set smaller. Generally, the default number of frame skipping intervals can be set to 2.

[0161] If the animal's posture indicates that it is defecating, the image of the person closest to the animal in the target image can be further detected and judged to determine whether the person in the target image has taken any action related to the animal's defecation behavior. Correspondingly, in practical applications, after the animal defecates, the animal's owner may also take some action; therefore, further detection and judgment of the person in the image can be performed to arrive at a final conclusion.

[0162] Specifically, for an animal detection box and multiple human detection boxes detected in a target image, the Euclidean distance between the animal and the human can be calculated, and the human with the smallest calculated Euclidean distance can be considered the animal's owner. Optionally, the Euclidean distance can be calculated based on the center coordinates of the detection box, or it can be calculated based on the coordinates of a fixed point within the detection box.

[0163] Specifically, after identifying the person closest to the animal in the target image, the person's image can be input into a model capable of recognizing the person's posture to determine their posture. The underlying principle is the same as the aforementioned scheme for behavior recognition of dogs.

[0164] Alternatively, the person image can be input into a model that can predict the key points of the person image, so that the model outputs the predicted key points of the person and determines the person's pose based on the predicted key points.

[0165] Among them, the key points of personnel can be marked by the skeletal points of the personnel, mainly including the head, limbs, torso and other parts. Figure 8a A schematic diagram of key points of a person provided in an embodiment of this application, such as... Figure 8a As shown, the key points are points 0-16 marked on the map, totaling 17 points.

[0166] If the detection and judgment are aimed at the behavior of bending over or squatting in the image, the skeletal points involved in these behaviors can be used as target key points, that is, the skeletal points of the lower limbs and torso are mainly used as target key points. Figure 8b This is a schematic diagram of target key points of a person provided in an embodiment of this application, such as... Figure 8b As shown, the eight points on the person's left shoulder, right shoulder, left hip, right hip, left knee, right knee, left ankle, and right ankle can be used as target key points.

[0167] Optionally, the person's posture can be determined based on the bending angles of the person's skeletal keypoints. Specifically, after the model that can predict the keypoints of a person's image outputs the predicted skeletal coordinates, the inverse cosine function can be used to calculate the inverse cosine of the skeletal coordinates to determine the bending angle, and the person's posture can be determined based on the calculated bending angle. For example, if the calculated shoulder-hip-knee angle is acute, the person is determined to be in a bent-over posture; if the calculated hip-knee-ankle angle is acute, the person is determined to be in a squatting posture.

[0168] For example, taking the calculation of key points on the same side of a person's torso as an example, if a person's posture changes from normal walking to squatting, the angle formed by the hip-knee-ankle bone points will change from an obtuse angle to an acute angle. The length of the three sides is calculated based on the triangle formed by the coordinates of these three bone points. Figure 9 This is a schematic diagram of a triangle formed by the coordinates of three skeletal points—hip, knee, and ankle—of a person, as provided in an embodiment of this application. Figure 9 As shown, where, Then, the inverse cosine function is used to calculate the bending angle at the corresponding point, such as... If the calculated bending angle is an acute angle, then the person is determined to be in a bent-over posture.

[0169] Step E3: If it is determined from the personnel's posture that the personnel are not cleaning up the animal's excrement, then it is determined that both the animal and the personnel have engaged in uncivilized behavior;

[0170] After determining the posture of the person in different images, the changes in the person's body posture can be determined based on the changes in the angles of key skeletal points in multiple images. Specifically, if the person's body posture did not change before the dog disappeared, it is assumed that the person did not clean up the animal's excrement while walking the dog, thus determining that both the animal and the person engaged in uncivilized behavior.

[0171] Step E4: If the behavior of cleaning up animal excrement is determined based on the posture of the personnel, then it is determined that there is no uncivilized behavior by either the animals or the personnel.

[0172] If a person's posture changes from normal walking to squatting or bending over, and then returns to normal walking after a certain period of time, it is considered that the person was cleaning up the animal's excrement while walking the dog, and therefore it is determined that neither the animal nor the person engaged in uncivilized behavior.

[0173] The technical solution of this application embodiment, for a sample image containing a target, improves the display of pixels by multiplying each pixel in a preset region of the sample image by a magnification factor, and adjusts the brightness, chroma, and saturation of each pixel in the preset region. In particular, in response to the possibility of overexposure or underexposure after multiplying the color space parameters by the preset magnification factor, this application solution also adaptively adjusts the brightness value of the preset region so that each pixel in the preset region reaches a more suitable brightness value. To a certain extent, this solves the problem of loss of key feature information caused by overexposure or underexposure of the sample image, realizes the enhancement of image data, and the new sample image obtained can also realize the expansion of the number of sample images and the improvement of quality, thereby improving the accuracy of model recognition and being able to cope with the recognition of various complex environments.

[0174] Figure 10 This is a schematic diagram of a target behavior detection device provided in an embodiment of this application. This device can execute the target behavior detection method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Figure 10 As shown, the device includes:

[0175] The sample expansion module 1010 is used to extract and reconstruct the target region of interest in a sample image containing a target, so as to expand the number of sample images; wherein, the sample image is labeled with key points and target pose;

[0176] Training module 1011 is used to train the initial model during the process of predicting key points in the expanded sample images using the initial model and during the process of classifying target poses based on the predicted key points.

[0177] The target pose recognition model determination module 1012 is used to end the training to obtain a target pose recognition model when the loss function meets the requirements, so as to identify the pose of the target in the target image based on the target pose recognition model, and determine the behavior of the target based on the pose of the target.

[0178] Adding features of the target to the preset area includes:

[0179] Target key points are determined from the key points marked in the sample image, and target regions of interest are determined based on the regions corresponding to the target key points; wherein, the number of target key points is less than or equal to the total number of key points;

[0180] The target region of interest is extracted from the sample image, and the target region of interest is added to the preset region to expand the number of sample images.

[0181] Determining the region of interest based on the region corresponding to the target key points includes:

[0182] For each target key point, determine a preset number of related key points that are closest to the target key point;

[0183] The smallest bounding polygon containing the target key points and related key points is taken as the target region of interest.

[0184] Add the target region of interest to the preset area, including:

[0185] The smallest bounding polygon of all key points in the sample image is taken as the global region of interest.

[0186] The region outside the global region of interest in the sample image is taken as the preset region, and the target region of interest is added to the preset region;

[0187] After adding the target region of interest to the preset region, the method further includes:

[0188] If the target region of interest extends beyond the sample image, the target region of interest is adjusted so that it lies within the sample image; the adjustment method includes at least one of scaling, translation, and rotation.

[0189] Adjusting the features of the preset region further includes:

[0190] Convert each pixel in the preset area from the RGB color space to the HSV color space;

[0191] Multiply the color space parameters in the HSV color space by a preset scaling factor to obtain the adjusted parameters in the HSV color space.

[0192] Convert each pixel in the preset area from the HSV color space back to the RGB color space.

[0193] The adjusted parameters include the adjusted brightness, chroma, and saturation;

[0194] After multiplying the color space parameters in the HSV color space by a preset scaling factor to obtain the adjusted parameters, the method further includes:

[0195] Determine the average brightness of a region within a preset area of ​​the sample image, and the average brightness of the image within the sample image;

[0196] If the average brightness of the region is within a first preset range and the average brightness of the region is greater than the average brightness of the image, or if the average brightness of the region is within a second preset range and the average brightness of the region is less than the average brightness of the image, then for each pixel in the preset region, the first exponent of the adjusted brightness of that pixel is taken as the final brightness of that pixel.

[0197] If the average brightness of the area is within the third preset range, then the brightness of the pixels in the preset area will not be adjusted.

[0198] If the average brightness of the region is within a first preset interval and the average brightness of the region is less than or equal to the average brightness of the image, or if the average brightness of the region is within a second preset interval and the average brightness of the region is greater than or equal to the average brightness of the image, then for each pixel in the preset region, the second exponent of the adjusted brightness of that pixel is taken as the final brightness of that pixel.

[0199] The first index is determined based on the ratio of the average brightness of the region to the average brightness of the image, and the second index is determined based on the ratio of the average brightness of the image to the average brightness of the region; the first preset interval is greater than the third preset interval, and the third preset interval is greater than the second preset interval.

[0200] The final chromaticity and final saturation of each pixel in the preset area are calculated based on the final brightness.

[0201] The process of predicting key points in the expanded sample images using the initial model includes:

[0202] In the process of extracting sample image features based on the self-attention mechanism, the maximum number of elements in each row of the initial similarity matrix are normalized, and the other elements are set to zero to obtain the final similarity matrix.

[0203] The process of determining the loss function in keypoint prediction of augmented sample images using the initial model includes:

[0204] Determine the first difference data between the labeled key points in the sample image and the predicted key points of the initial model;

[0205] Determine the second difference data between the labeled key points in the sample image and the predicted labeled key points that have the greatest impact on the labeled key points;

[0206] The loss function is determined based on the first difference data and the second difference data.

[0207] After obtaining the target pose recognition model by ending training when the loss function meets the requirements, the method further includes:

[0208] The animal image in the target image is input into the target pose recognition model to determine the animal pose;

[0209] If it is determined that the animal is defecating based on its posture, then the image of the person closest to the animal in the target image is input into the target posture recognition model to determine the person's posture.

[0210] If it is determined from the person's posture that the person is not cleaning up the animal's excrement, then it is determined that both the animal and the person are engaging in uncivilized behavior.

[0211] If it is determined from the person's posture that the person is cleaning up the animal's excrement, then it is determined that neither the animal nor the person is engaging in uncivilized behavior.

[0212] The process of determining the target image includes:

[0213] Target detection and tracking of animals in the acquired images are performed, and the overlap of detection boxes of the same animal in adjacent frames of acquired images is calculated.

[0214] If the overlap data is greater than the preset overlap data, then the images captured in adjacent frames will be used as the target image.

[0215] The process of determining whether an animal is exhibiting defecation behavior based on its posture includes:

[0216] If the ratio of the number of target images whose animal postures match the preset animal postures to the total number of target images reaches the preset ratio, then it is determined that the animal is exhibiting excretion behavior.

[0217] The technical solution of this application embodiment, for a sample image containing a target, adjusts the features of a preset region in the sample image to obtain a new sample image. The new sample image and the initial sample image are used together as a sample image to expand the number of sample images. The initial model is trained during the process of predicting key points in the expanded sample image using the initial model, and during the process of classifying the target pose based on the predicted key points. Training ends when the loss function meets the requirements, resulting in a target pose recognition model. This model is then used to identify the pose of the target in the target image and determine the target's behavior based on the pose. This solution, by adjusting the features in a preset region of the sample image, including adding target features to the preset region, to obtain a new sample image, achieves both quantity expansion and quality improvement of the sample images, thereby improving the accuracy of model recognition and enabling the recognition of various complex environments.

[0218] The target behavior detection device provided in this application embodiment can execute a target behavior detection method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0219] Figure 11 A schematic diagram of an electronic device 1110 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0220] like Figure 11 As shown, the electronic device 1110 includes at least one processor 1111 and a memory, such as a read-only memory (ROM) 1112 or a random access memory (RAM) 1113, connected to the at least one processor 1111 for data processing. The memory stores computer programs executable by the at least one processor. The processor 1111 can perform various appropriate actions and processes based on the computer program stored in the ROM 1112 or loaded into the RAM 1113 from storage unit 1118. The RAM 1113 may also store various programs and data required for the operation of the electronic device 1110. The processor 1111, ROM 1112, and RAM 1113 are interconnected via a bus 1114. An input / output (I / O) interface 1115 is also connected to the bus 1114.

[0221] Multiple components in electronic device 1110 are connected to I / O interface 1115, including: input unit 1116, such as keyboard, mouse, etc.; output unit 1117, such as various types of monitors, speakers, etc.; storage unit 1118, such as disk, optical disk, etc.; and data processing unit 1119, such as network card, modem, wireless data processing transceiver, etc. Data processing unit 1119 allows electronic device 1110 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0222] Processor 1111 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 1111 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 1111 performs the various methods and processes described above, such as target behavior detection methods.

[0223] In some embodiments, the target behavior detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 1118. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 1110 via (ROM) 1112 and / or data processing unit 1119. When the computer program is loaded into (RAM) 1113 and executed by processor 1111, one or more steps of the target behavior detection method described above may be performed. Alternatively, in other embodiments, processor 1111 may be configured to perform the target behavior detection method by any other suitable means (e.g., by means of firmware).

[0224] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0225] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable target behavior detection device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0226] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0227] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0228] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected through digital data processing (e.g., data processing networks) of any form or medium. Examples of data processing networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0229] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via data processing networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0230] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0231] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target behavior detection method, characterized in that, The method includes: For a sample image containing a target, the features of a preset region in the sample image are adjusted to obtain a new sample image. The new sample image and the initial sample image are used together as a sample image to expand the number of sample images. The adjustment of the features of the preset region includes adding the features of the target to the preset region; the sample image is labeled with key points and the target pose; The initial model is trained during the process of predicting key points in the expanded sample images using the initial model and during the process of classifying target poses based on the predicted key points. Training ends when the loss function meets the requirements, resulting in a target pose recognition model. This model is used to identify the pose of a target in a target image and determine the target's behavior based on that pose.

2. The method according to claim 1, characterized in that, Adding features of the target to the preset area includes: Target key points are determined from the key points marked in the sample image, and target regions of interest are determined based on the regions corresponding to the target key points; wherein, the number of target key points is less than or equal to the total number of key points; The target region of interest is extracted from the sample image, and the target region of interest is added to the preset region to expand the number of sample images.

3. The method according to claim 2, characterized in that, Determining the region of interest based on the region corresponding to the target key points includes: For each target key point, determine a preset number of related key points that are closest to the target key point; The smallest bounding polygon containing the target key points and related key points is taken as the target region of interest.

4. The method according to claim 1, characterized in that, Adjusting the features of the preset region further includes: Convert each pixel in the preset area from the RGB color space to the HSV color space; Multiply the color space parameters in the HSV color space by a preset scaling factor to obtain the adjusted parameters in the HSV color space. Convert each pixel in the preset area from the HSV color space back to the RGB color space.

5. The method according to claim 4, characterized in that, The adjusted parameters include the adjusted brightness, chroma, and saturation; After multiplying the color space parameters in the HSV color space by a preset scaling factor to obtain the adjusted parameters, the method further includes: Determine the average brightness of a region within a preset area of ​​the sample image, and the average brightness of the image within the sample image; If the average brightness of the region is within a first preset range and the average brightness of the region is greater than the average brightness of the image, or if the average brightness of the region is within a second preset range and the average brightness of the region is less than the average brightness of the image, then for each pixel in the preset region, the first exponent of the adjusted brightness of that pixel is taken as the final brightness of that pixel. If the average brightness of the area is within the third preset range, then the brightness of the pixels in the preset area will not be adjusted. If the average brightness of the region is within a first preset interval and the average brightness of the region is less than or equal to the average brightness of the image, or if the average brightness of the region is within a second preset interval and the average brightness of the region is greater than or equal to the average brightness of the image, then for each pixel in the preset region, the second exponent of the adjusted brightness of that pixel is taken as the final brightness of that pixel. The first index is determined based on the ratio of the average brightness of the region to the average brightness of the image, and the second index is determined based on the ratio of the average brightness of the image to the average brightness of the region; the first preset interval is greater than the third preset interval, and the third preset interval is greater than the second preset interval. The final chromaticity and final saturation of each pixel in the preset area are calculated based on the final brightness.

6. The method according to claim 1, characterized in that, The process of predicting key points in the expanded sample images using the initial model includes: In the process of extracting sample image features based on the self-attention mechanism, the maximum number of elements in each row of the initial similarity matrix are normalized, and the other elements are set to zero to obtain the final similarity matrix.

7. The method according to claim 1, characterized in that, The process of determining the loss function in keypoint prediction of augmented sample images using the initial model includes: Determine the first difference data between the labeled key points in the sample image and the predicted key points of the initial model; Determine the second difference data between the labeled key points in the sample image and the predicted labeled key points that have the greatest impact on the labeled key points; The loss function is determined based on the first difference data and the second difference data.

8. A target behavior detection device, characterized in that, The device includes: The sample expansion module is used to extract and reconstruct the target region of interest in a sample image containing a target, so as to expand the number of sample images; wherein, the sample image is labeled with key points and target pose; The training module is used to train the initial model during the process of predicting key points in the expanded sample images using the initial model and during the process of classifying target poses based on the predicted key points. The target pose recognition model determination module is used to terminate training and obtain a target pose recognition model when the loss function meets the requirements, so as to identify the pose of the target in the target image based on the target pose recognition model, and determine the behavior of the target based on the pose of the target.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and The memory is connected to the at least one processor for data processing; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the target behavior detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the target behavior detection method according to any one of claims 1-7.

Images