A method, system and device for identifying target behavior

Abstract

The present application relates to the field of image processing technology, and discloses a method, system and device for identifying target behavior, wherein the method includes obtaining a video stream to be processed, wherein the video frame of the video stream includes a target object to be identified; based on the target object identified in the video frame, the video frame is divided into one or more areas to be detected; the skeleton key point heat map features of the area to be detected are extracted, and the spatial features of the area to be detected are obtained by convolution; the skeleton key point heat map features and the spatial features are superimposed to obtain superimposed features, and the superimposed features are shifted in the time dimension to obtain fused features with time series information; and whether the target behavior occurs in the video frame is determined according to the fused features. The technical solution provided by one or more embodiments of the present disclosure can improve the accuracy of target behavior recognition.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, system and device for identifying target behavior. Background Art

[0002] In the related technology, single-frame target detection is usually performed. Specifically, skeleton key points can be obtained, the instantaneous motion rate of the target object and the distance between related target objects can be calculated, and whether the target behavior occurs can be determined by the crowd density and the moving speed of the characters.

[0003] However, in some target behavior recognition scenarios, key points may be lost due to overlap and occlusion of the target objects, making it impossible to judge the continuity of the action, which can easily lead to false detection or missed detection, resulting in low accuracy of target behavior recognition.

[0004] In view of this, a more accurate target behavior recognition method is currently needed. Summary of the invention

[0005] One or more embodiments of the present disclosure provide a method, system, and device for identifying a target behavior, which can improve the accuracy of target behavior identification.

[0006] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0007] On the one hand, the present disclosure provides a method for identifying a target behavior, the method comprising: obtaining a video stream to be processed, wherein a video frame of the video stream includes a target object to be identified; based on the target object identified in the video frame, dividing the video frame into one or more areas to be detected; extracting skeleton key point heat map features of the area to be detected, and obtaining spatial features of the area to be detected obtained by convolution; superimposing the skeleton key point heat map features and the spatial features to obtain superimposed features, and shifting the superimposed features in the time dimension to obtain fused features with time series information; judging whether the target behavior occurs in the video frame according to the fused features.

[0008] On the one hand, the present disclosure provides a target behavior recognition system, which includes: an image receiving unit, used to receive a video stream to be processed, wherein the video stream includes multiple target objects to be recognized; a privacy protection unit, used to perform privacy processing on video frames in the video stream to obtain privacy-protected target video frames; an adaptive region selection unit, used to determine position information of each target object and divide the video frame into one or more regions to be detected; an image super-resolution unit, used to perform super-resolution processing on the region to be detected; and a target behavior recognition unit, used to recognize an image in the region to be detected to obtain a behavior recognition result of the target object in the region to be detected.

[0009] On the other hand, the present disclosure further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above-mentioned target behavior is recognized.

[0010] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned target behavior recognition is realized.

[0011] The technical solution provided by one or more embodiments of the present disclosure can determine whether the target behavior occurs based on the input video stream. Specifically, first obtain the video stream of the target object, determine the area to be detected in the video frame of the video stream, input the area to be detected into the model for detection, and obtain the video stream detection result. Obtain the audio stream to detect keywords and obtain the audio stream detection result. Determine whether the target behavior occurs based on the video stream detection result and the audio stream detection result.

[0012] It can be seen that the technical solution provided by the present disclosure can improve the accuracy and comprehensiveness of target behavior recognition. It not only does not rely on the detection accuracy of key points, but also does not need to process a large amount of irrelevant information and background noise, which greatly improves the accuracy of recognition. At the same time, the system performs privacy processing and super-resolution processing on the target objects in the detection area, which improves the privacy of the system and the recognition efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1A schematic diagram showing the steps of a target behavior recognition method in one embodiment of the present disclosure is shown;

[0015] Figure 2 A schematic diagram showing selection of an area to be detected in an application scenario of the present disclosure is shown;

[0016] Figure 3 A schematic diagram showing the steps of a method for dividing a region to be detected in one embodiment of the present disclosure is shown;

[0017] Figure 4 A schematic diagram showing the heat map features of key points of the head in an application scenario of the present disclosure is shown;

[0018] Figure 5 A schematic diagram showing a feature fusion process in an application scenario of the present disclosure is shown;

[0019] Figure 6 A schematic diagram showing the training steps of a target behavior recognition model in one embodiment of the present disclosure is shown;

[0020] Figure 7 A schematic diagram of functional modules of a target behavior system in one embodiment of the present disclosure is shown;

[0021] Figure 8 A schematic structural diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0023] With the continuous development of deep learning technology, related technologies can determine the skeleton position information of the target object by identifying skeleton key points, and judge whether the target behavior occurs according to the instantaneous movement speed of some skeleton points; related technologies can also obtain the motion characteristics of the target object by convolution and time shifting the image through video classification. However, in some application scenarios, if target recognition is based on skeleton key points, some key points cannot be accurately detected when the target object is blocked or overlapped in the video, affecting the detection accuracy; if based on video classification technology, when the target object moves quickly, it is easy to produce motion blur and motion misjudgment, which in turn affects the detection accuracy.

[0024] In view of this, the present invention discloses one or more embodiments, which introduces the heat map features of the skeleton key points of the target object, fuses them with the temporal features and spatial features, and determines whether the target behavior occurs based on the fused features. In this way, on the one hand, the problem of high dependence on key point detection accuracy can be solved, and the misjudgment of actions due to environmental factors such as occlusion and overlap can be avoided. On the other hand, the continuity of the action can be captured to improve the detection accuracy.

[0025] Based on the above ideas, if all the images detected in the video are input into the model, they usually contain a lot of irrelevant information and background noise, which will interfere with the model's judgment and reduce the accuracy of detection. When processing the entire image, the model may not be able to focus on the key areas, resulting in inaccurate recognition of the target behavior. In addition, unprocessed videos are prone to leaking personal identity information and infringing on personal privacy. Therefore, how to process input images that meet these conditions is also a current technical problem.

[0026] In view of this, one or more embodiments of the present disclosure can process the input video stream to more accurately identify whether there is a target behavior. Figure 1 In one embodiment of the present disclosure, a method for identifying a target behavior is provided. The method is performed according to the following steps:

[0027] S1: Obtain a video stream to be processed, wherein the video frames of the video stream include a target object to be identified.

[0028] In this embodiment, the video stream may be video data collected by a camera, and the camera may be located in a distributed system architecture with two parts, in which, in addition to an edge end for collecting and preprocessing data, a cloud for processing and analyzing data may also be included. Continuous video frames are extracted from the input video stream, and the video frames contain one or more target objects to be identified, and the number of target objects to be identified is not limited.

[0029] It should be noted that in this embodiment, in order to ensure that personal information in the image is not leaked, such as sensitive information such as facial features, privacy processing is performed on the training image when collecting training image data for model training.

[0030] Optionally, in one implementation, privacy processing may be performed in the following manner:

[0031] S101: Detecting position information of a target object in the video frame, and determining a target range including the target object in the video frame based on the detected position information.

[0032] In this embodiment, a variety of methods can be used to detect the position of the target object. Taking face detection as an example, the face can be detected by the MTCNN algorithm, and the specific range can be determined based on the coordinates of the smallest rectangular frame including the face and the coordinates of the five key points of the face.

[0033] S103: Dynamically adjust the size and standard deviation of the Gaussian blur kernel according to the size of the target range. In addition, the size of the target range and the size of the Gaussian kernel are determined by the following formula:

[0034]

[0035]

[0036] Among them, the target range takes the face as an example. is the given target size, is the minimum face size, is the maximum face size, is the normalized face size, is the minimum kernel size, is the maximum kernel size.

[0037] S105: blurring the video frame according to the dynamically adjusted Gaussian blur kernel, and sending the blurred video frame to the cloud for recognition.

[0038] In this embodiment, the two-dimensional Gaussian blur kernel is defined as , the input image is , the blurred image is , the blurred image can be obtained by convolving the input image, which is calculated as follows:

[0039]

[0040]

[0041]

[0042] in, is the Gaussian standard deviation, is the minimum standard deviation, is the maximum standard deviation, are the coordinates of the pixel in the kernel relative to the center.

[0043] S2: Based on the target object identified in the video frame, divide the video frame into one or more areas to be detected.

[0044] In this embodiment, a video frame usually contains multiple target objects and a large amount of irrelevant information, which requires a lot of data processing, which may reduce the recognition rate and recognition accuracy. In view of this, the video frame can be divided into one or more areas to be detected, each of which contains at least two target objects. Figure 2 A video frame contains multiple target objects, where the target objects are located in different locations. According to the degree of aggregation of the target objects, the video frame is divided into six detection areas A, B, C, D, E, F, and G. Each area contains multiple target objects to be detected. In addition, when a target object appears alone, the area where the target object is located is not divided.

[0045] Specifically, please refer to Figure 3 , the area to be detected is determined as follows:

[0046] S201: Using a target tracking algorithm to identify a target object in a video frame, recording the position, identification and movement distance of the target object, and if the movement distance of the target object within a specified time is less than a set threshold, adding the position and identification to a status list.

[0047] In this embodiment, a target tracking algorithm is used to identify the target object in the video frame, record the position and identification of each target object, set a time value T and a distance value , record the distance moved by each target object within T time, if the distance moved by the target object within T time is less than the preset distance value , then the location and identification information of the target object is added to the status list.

[0048] S203: Acquire location information of a first target object in the status list and a second target object closest to the first target object to form a gathering area.

[0049] In this embodiment, the first target object in the state list is first obtained, the location information of the first target object is obtained, the state list is traversed to find the second target object that is closest to the first target object and whose distance is less than the set threshold d, and the location information of the second target object is obtained. The initial clustering area is formed with the first target object and the second target object as the boundary. And calculate The minimum circumscribed circle of is calculated as follows:

[0050]

[0051]

[0052] in, is the center of the smallest circumscribed circle, is the radius of the minimum circumscribed circle, ( , ) is the center point coordinate of the first target object, ( , ) are the center point coordinates of the second target object.

[0053] S205: Expand the gathering area, traverse other target objects in the status list, and if the distance from the center point of other target objects to the minimum circumscribed circle of the gathering area is less than a preset value, the target object is added to the gathering area and the minimum circumscribed circle is updated.

[0054] In this embodiment, for other target objects in the state list, the coordinates of their center points are obtained ( , ) and calculate its distance to the minimum circumscribed circle of the current gathering area ), calculated as:

[0055]

[0056] like ) is less than , then add the target object to the current gathering area and recalculate the minimum circumscribed circle of the current gathering area. The center of the circle is calculated using the Welzl algorithm, and the radius R is the radius of the minimum circumscribed circle that contains all the target objects in the current gathering area.

[0057] S207: After the traversal is completed, the minimum bounding rectangle in the clustered area is calculated, and the output rectangular area is the area to be detected. If the resolution of the area to be detected is lower than the preset resolution, the area to be detected is super-reconstructed.

[0058] In this embodiment, all target objects are traversed as described in S201 to S205 until the remaining target objects in the video frame cannot be included in a certain clustering area, and the minimum circumscribed rectangle of all clustering areas is calculated to form a region to be detected.

[0059] It should be noted that in the above process, the size and resolution of the output area to be detected are different, so it is necessary to perform resolution processing on the image of the area to be detected. The low-resolution image is enlarged to a fixed size, and features are extracted from the enlarged low-resolution image, and a high-resolution image is reconstructed based on the extracted features. Among them, the reconstruction process can use a variety of methods, such as interpolation method, sparse representation method, etc., which are not limited in this disclosure.

[0060] Through the above method, a large number of video frames included in the input video stream can be processed, and a large amount of irrelevant information and environmental noise can be excluded through adaptive area division, reducing the difficulty of image processing. In addition, privacy processing can be applied to the input video frames to protect personal information.

[0061] S3: Extract the skeleton key point heat map features of the area to be detected, and obtain the spatial features of the area to be detected after convolution.

[0062] In this embodiment, the heat map features of skeleton key points are used as supervision signals, position information is introduced, 2D convolution is applied separately to the area to be detected in each video frame, spatial features are extracted, and a spatial feature map is output. In practical applications, occlusion and overlap are prone to occur between multiple target objects. If only skeleton key points are identified, some key points will be lost. Therefore, in this embodiment, the heat map features of skeleton key points are used, and the pixel values ​​of the heat map are used to represent the probability that the position is a key point, and the probability is usually distinguished by brightness or color intensity. It should be noted that the skeleton key point heat map is a density map used to represent the possible positions of human key points in an image, and each key point has a corresponding heat map feature.

[0063] Specifically, the skeleton key point heat map features are obtained according to the following method:

[0064] S301: Acquire the coordinates of skeleton key points contained in the area to be detected, and generate a corresponding heat map on the coordinates of one or more key points, wherein the heat map represents the thermal probability distribution of the key points in the area to be detected, wherein the generation of the heat map is determined according to the Gaussian distribution generated on the coordinates;

[0065] In this embodiment, the present disclosure does not limit the identification of skeleton key points. For example, the mscoco model can be used to identify the key points of multiple target objects and obtain the coordinates of the key points, where a target object contains a total of 17 key points, and a corresponding heat map is generated at the coordinates of each key point.

[0066] In one embodiment, if the input image is a 320×320 RGB color image, and the coordinates of the key points of the head of a target object contained in the image are (160, 100), then the size of the corresponding heat map output by the head is 320×320. Figure 4 In this head key point heat map, the probability value at the coordinate (160, 100) is about 0.9. As the distance from this coordinate point increases, the probability value will gradually decrease, forming a Gaussian distribution.

[0067] S303: Obtain a heat map of the one or more skeleton key points, and superimpose them on positions corresponding to the key points in the area to be detected to form a skeleton key point heat map and extract features of the skeleton key point heat map.

[0068] In the above manner, a continuous probability distribution can be provided through the skeleton key point heat map to determine the position information of the key points of the target object, thereby more accurately representing and detecting the position of the human joints and avoiding misjudgment caused by occlusion and overlap.

[0069] S4: superimposing the skeleton key point heat map feature and the spatial feature to obtain a superimposed feature, and performing a shift operation on the superimposed feature in the time dimension to obtain a fusion feature with time series information.

[0070] See also Figure 5 In this embodiment, the position information of the skeleton key points is combined with the spatial features of the image, and the temporal features are obtained. Specifically, the skeleton key point heat map shows the position and activity intensity of the key parts of the human body in the image, and the spatial feature map contains spatial information such as the texture, edge and shape of the image. The skeleton key point heat map is superimposed with the spatial feature map to obtain a superimposed feature map. TSM (Temporal Shift Module) is used to cyclically shift the feature channel in the time dimension to capture the timing information and dynamic changes of the action, and obtain fusion features with timing information.

[0071] Through the above method, fusion features with temporal information are obtained. These fusion features not only contain the information of a single frame image, but also integrate the dynamics between video frames, providing support for subsequent behavior recognition and action analysis.

[0072] S5: Determine whether the target behavior occurs in the video frame according to the fusion feature.

[0073] In this embodiment, the fused features obtained in S4 are input into a classifier or a detection model for identification to determine whether the target behavior occurs. For example, a support vector machine or a convolutional neural network can be used to identify the fused features.

[0074] In the above manner, the video frames of the input video stream can be processed to obtain fusion features containing position information and timing information, and the fusion features can be used to determine whether the target behavior occurs, thereby obtaining the judgment result of the video stream. However, some target behaviors are not only manifested as physical movements, such as fighting, but also accompanied by verbal quarrels and threats. These verbal behaviors cannot be captured by visual information alone. Therefore, the present invention discloses an environmental sound recognition method, which detects the target behavior by detecting the environmental volume and sensitive words. The method is as follows:

[0075] An audio stream matching the video stream is obtained, and whether there are keywords associated with the target behavior in the audio stream is identified. The identification result of the audio stream is combined with the judgment result of the video stream to obtain a final recognition result of the target behavior.

[0076] In this embodiment, after judging multiple video frames and obtaining the judgment result of the video stream, if the target behavior exists, the audio data matching these video frames can be obtained for further judgment. Specifically, the audio data of the same time period is synchronized and extracted according to the timestamp of the video frame, and the audio data is parsed to determine whether there is an audio feature related to the target behavior. If it exists, it is determined that the target behavior currently exists. If not, it is considered whether there is a misjudgment, and further judgment is required based on more data.

[0077] Also, see Figure 6 The present invention discloses a target behavior recognition model that can analyze fusion features, which can improve the accuracy of recognition. The training method of the model is as follows:

[0078] S501: Acquire a training image, where the training image is a privacy-processed image.

[0079] In this embodiment, in order to ensure that personal information in the image is not leaked, such as sensitive information such as facial features, privacy processing is performed on the training image when collecting training image data for model training.

[0080] Optionally, in one implementation, privacy processing may be performed in the manner described in S101 to S103.

[0081] Optionally, in one implementation, privacy processing may also be performed in the following manner:

[0082] S111: Divide the model into a shallow feature extraction network and a deep inference network for deployment.

[0083] S113: deploying a shallow feature extraction network of the model at the edge, quantizing the edge model, and extracting features from the quantized model to obtain a feature tensor.

[0084] S115: deploying the deep inference network of the model in the cloud, sending the feature tensor to the cloud server for dequantization processing, and using the deep inference network to infer the model.

[0085] In this embodiment, the model is divided into two parts for remote deployment. The first part is from the input of the original model to a pre-trained layer of the network, and the second part is from the network to the output of the original model. The intermediate inference results are transmitted through the network. Since the content transmitted by the network is the feature vector after shallow network feature extraction, the original image data does not exist, thereby achieving the purpose of privacy protection.

[0086] S503: extracting a heat map of skeleton key points of the training image, and superimposing the heat map with the spatial feature map of the training image to obtain a superimposed feature, and performing a time shift operation on the superimposed feature to obtain a fused feature.

[0087] In this embodiment, the skeleton key point heat map and the spatial feature map can be extracted and processed according to the method of S3 to S4, and the fusion feature can be obtained, which will not be described in detail here.

[0088] S505: Input the fused features into a fully connected layer for classification, and the fully connected layer converts the fused features into a first score of the target behavior and a second score of the non-target behavior.

[0089] In this embodiment, the fully connected layer converts the fused features into classification scores. Specifically, the input features are mapped to the categories of predefined target behaviors, and the scores of each category are calculated, where the highest score is usually considered to be the predicted category. The scores are calculated by each neuron in the fully connected layer through weights and biases, which are optimized by the back propagation algorithm during the training process.

[0090] S507: Convert the first score into a probability value of the target behavior through a softmax function, and convert the second score into a probability value of a non-target behavior.

[0091] In this embodiment, the score is usually converted into a probability distribution through an activation function. Taking the softmax function as an activation function as an example, the score of each category represents the probability that the input data belongs to that category. The first score of the target behavior is represented by the probability that the model believes that the input data belongs to the target category, and the non-target behavior is represented by the probability that the input data belongs to other categories.

[0092] S509: Calculate the loss between the probability value and the preset label using the loss function, calculate the gradient of the loss function with respect to the network parameters, and update the weight of the network.

[0093] In this embodiment, in order to evaluate the performance of the model, a cross entropy loss function is usually used to calculate the difference between the probability value output by the model and the preset label, and the gradient of the loss function with respect to the network parameters is calculated by the back propagation algorithm, and the weight of the network is updated. After multiple iterations, the prediction of the model is closer to the preset label, thereby improving the accuracy of the model prediction.

[0094] See also Figure 7 , an embodiment of the present disclosure further provides a target behavior recognition system, the system comprising:

[0095] The image receiving unit 100 is used to receive a video stream to be processed, wherein the video stream includes a plurality of target objects to be identified.

[0096] The privacy protection unit 200 is used to perform privacy processing on the video frames in the video stream to obtain a target video frame after privacy protection.

[0097] The adaptive region selection unit 300 is used to determine the position information of each target object and divide the video frame into one or more regions to be detected.

[0098] The image super-resolution unit 400 is used to perform super-resolution processing on the area to be detected.

[0099] The target behavior recognition unit 500 is used to recognize the image in the area to be detected and obtain a behavior recognition result of the target object in the area to be detected.

[0100] Among them, the specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, which will not be repeated here.

[0101] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8A processor 10 is taken as an example.

[0102] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0103] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0104] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0106] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0107] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0108] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0109] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

[0110] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0111] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to the computer device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0112] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the computer device.

[0113] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0114] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0115] It is understandable that in the specific implementation of the present application, related data such as user information, location information, navigation data, etc. are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0116] The devices or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0117] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0118] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0119] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0123] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0125] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for identifying a target behavior, characterized in that: The method comprises: Acquire a video stream to be processed, wherein the video frames of the video stream include a target object to be identified; Based on the target objects identified in the video frame, dividing the video frame into one or more areas to be detected according to the degree of aggregation of the target objects, wherein the areas to be detected contain at least two or more of the target objects; Extracting the skeleton key point heat map features of the area to be detected, and obtaining the spatial features of the area to be detected obtained by convolution; Superimposing the skeleton key point heat map feature and the spatial feature to obtain a superimposed feature, and performing a shift operation on the superimposed feature in the time dimension to obtain a fusion feature with time series information; Whether a target behavior occurs in the video frame is determined according to the fusion feature.

2. The method according to claim 1, characterized in that: The method further comprises: Acquire an audio stream that matches the video stream, and identify whether there are keywords associated with the target behavior in the audio stream; The recognition result of the audio stream is combined with the judgment result of the video stream to obtain the final recognition result of the target behavior.

3. The method according to claim 1, characterized in that After obtaining the video stream to be processed, the method further includes: Detecting position information of a target object in the video frame, and determining a target range including the target object in the video frame based on the detected position information; Dynamically adjust the size and standard deviation of the Gaussian blur kernel according to the size of the target range; The video frame is blurred according to the dynamically adjusted Gaussian blur kernel, and the blurred video frame is sent to the cloud for recognition.

4. The method according to claim 3, characterized in that The method further comprises: The model is divided into a shallow feature extraction network and a deep reasoning network for deployment; Deploy a shallow feature extraction network of the model at the edge, quantize the edge model, and extract features from the quantized model to obtain a feature tensor; The deep inference network of the model is deployed in the cloud, the feature tensor is sent to the cloud server for dequantization processing, and the model is inferred using the deep inference network.

5. The method according to claim 4, characterized in that The target behavior recognition model is trained in the following manner: Acquire a training image, where the training image is a privacy-processed image; Extracting a skeleton key point heat map of the training image and superimposing it with the spatial feature map of the training image to obtain a superimposed feature, and performing a time shift operation on the superimposed feature to obtain a fused feature; Inputting the fused features into a fully connected layer for classification, the fully connected layer converting the fused features into a first score of the target behavior and a second score of the non-target behavior; Converting the first score into a probability value of a target behavior through a softmax function, and converting the second score into a probability value of a non-target behavior; The loss function is used to calculate the loss between the probability value and the preset label, the gradient of the loss function with respect to the network parameters is calculated, and the weight of the network is updated.

6. The method according to claim 1 or 5, characterized in that: The method for acquiring the skeleton key point heat map feature includes: Obtaining coordinates of skeleton key points contained in the area to be detected, generating corresponding heat maps on the coordinates of one or more key points, wherein the heat maps represent the thermal probability distribution of the key points in the area to be detected, wherein the generation of the heat maps is determined according to the Gaussian distribution generated on the coordinates; The heat map of the one or more skeleton key points is obtained, and superimposed on the positions corresponding to the key points in the area to be detected to form a skeleton key point heat map and extract the skeleton key point heat map features.

7. The method according to claim 1, characterized in that Dividing the video frame into one or more areas to be detected, the method comprising: Use a target tracking algorithm to identify the target object in the video frame, record the position, identification and movement distance of the target object, and if the movement distance of the target object within a specified time is less than a set threshold, add the position and identification to the status list; Acquire location information of a first target object in the status list and a second target object closest to the first target object to form a gathering area; Expand the gathering area, traverse other target objects in the state list, and if the distance from the center point of other target objects to the minimum circumscribed circle of the gathering area is less than a preset value, then the target object is added to the gathering area and the minimum circumscribed circle is updated; After the traversal is completed, the minimum bounding rectangle in the clustered area is calculated, and the output rectangular area is the area to be detected. If the resolution of the area to be detected is lower than the preset resolution, the area to be detected is super-reconstructed.

8. The method according to claim 7, characterized in that The minimum circumscribed circle and the distance from the current target to the minimum circumscribed circle are calculated as follows: The initial minimum circumscribed circle is a circular area with the distance between the first target object and the second target object as its diameter, and the location information of the current target is obtained. , the minimum circumscribed circle distance from the current target to the current area for: in, is the center coordinate of the minimum circumscribed circle of the current region. After the current target is added to the current region, the minimum circumscribed circle is recalculated. The minimum circumscribed circle should be the circumscribed circle that contains all target objects in the current region.

9. A target behavior recognition system, characterized in that: The system comprises: An image receiving unit, configured to receive a video stream to be processed, wherein the video stream includes a plurality of target objects to be identified; A privacy protection unit, configured to perform privacy processing on the video frames in the video stream to obtain a target video frame that has undergone privacy protection; An adaptive area selection unit, used to determine the position information of each target object, and divide the video frame into one or more to-be-detected areas according to the concentration degree of the target objects, wherein the to-be-detected areas contain at least two or more of the target objects; An image super-resolution unit, used for performing super-resolution processing on the area to be detected; The target behavior recognition unit is used to recognize the image in the area to be detected to obtain a behavior recognition result of the target object in the area to be detected.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 9 by executing the computer instructions.

Images