Key point detection method, behavior recognition method, device and readable storage medium

Abstract

The application provides a key point detection method, a behavior recognition method, a behavior recognition device and a computer readable storage medium. The key point detection method comprises: extracting output feature maps of different down-sampling resolutions of a to-be-detected image by using a plurality of re-parameterized convolution blocks stacked in sequence; up-sampling the output feature maps of different down-sampling resolutions, and fusing the output feature maps of the same resolution after up-sampling until a final fused feature map is obtained; and obtaining coordinate points with pixel values satisfying a preset condition from the fused feature map as key points in the to-be-detected image. In the above manner, the behavior recognition device uses the structure re-parameterization idea to construct a key point detection network composed of re-parameterized convolution blocks stacked, improves the precision and performance of the key point detection network, reduces the parameter quantity and calculation quantity of the key point detection network, and accelerates the inference speed of the key point detection network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of key point detection, in particular to a key point detection method, a behavior recognition method, a behavior recognition device and a computer readable storage medium. BACKGROUND

[0002] In recent years, China is rapidly entering an aging society. With the acceleration of the aging process, the smart elderly care industry has developed rapidly, and the dumping behavior of the elderly is a core problem of smart elderly care.

[0003] The core problem of recognizing the dumping behavior of the elderly is how to accurately identify the human key points of the elderly. However, the current key point detection model has a large amount of calculation and a complex model structure, which leads to a large amount of algorithm calculation and slow detection speed, and poor real-time performance. SUMMARY

[0004] The present application provides a key point detection method, a behavior recognition method, a behavior recognition device and a computer readable storage medium.

[0005] The present application provides a key point detection method, which comprises:

[0006] Using a plurality of sequentially stacked heavy parameterization convolution blocks to extract output feature maps of different down-sampling resolutions of the to-be-detected image;

[0007] Up-sampling the output feature maps of different down-sampling resolutions, and fusing the output feature maps of the same resolution after up-sampling until obtaining a final fused feature map;

[0008] Obtaining coordinate points with pixel values satisfying a preset condition from the fused feature map as key points in the to-be-detected image.

[0009] Among them, the output feature maps of different down-sampling resolutions include a first output feature map, a second output feature map, a third output feature map and a fourth output feature map in descending order of resolution;

[0010] The up-sampling of the output feature maps of different down-sampling resolutions, and the fusion of the output feature maps of the same resolution after up-sampling until obtaining a final fused feature map, comprises:

[0011] Up-sampling the fourth output feature map, and fusing the up-sampled fourth output feature map with the third output feature map to obtain a fifth output feature map;

[0012] Up-sampling the fifth output feature map, and fusing the up-sampled fifth output feature map with the second output feature map to obtain a sixth output feature map;

[0013] The sixth output feature map is up-sampled, and the up-sampled sixth output feature map is fused with the first output feature map to obtain the fused feature map.

[0014] The pixel value of the coordinate point in the fused feature map satisfying a preset condition is obtained as a key point in the to-be-detected image.

[0015] The coordinate point of the maximum pixel value in the fused feature map is obtained as a key point in the to-be-detected image.

[0016] The pixel value of the coordinate point in the fused feature map satisfying a preset condition is obtained as a key point in the to-be-detected image.

[0017] Based on the down-sampling relationship between the fused feature map and the to-be-detected image, an offset pixel value of a quantization error is obtained.

[0018] According to the offset pixel value, the coordinate point is offset processed according to an activation direction to obtain an offset coordinate point.

[0019] According to the down-sampling relationship, the offset coordinate point is mapped to an original coordinate space of the to-be-detected image to obtain a key point in the to-be-detected image.

[0020] The activation direction is a direction from the coordinate point of the maximum pixel value to a coordinate point of a second maximum pixel value.

[0021] The application also provides a behavior recognition method, which comprises:

[0022] Obtaining a plurality of to-be-detected images.

[0023] Obtaining a plurality of human key points of a target object in each to-be-detected image by using the key point detection method.

[0024] Obtaining the barycentric position information of the target object in each to-be-detected image according to the plurality of human key points of each to-be-detected image.

[0025] Obtaining the barycentric descent speed of the target object by using the barycentric position information of the target object in a plurality of to-be-detected images.

[0026] When the barycentric descent speed is greater than a first preset threshold, it is determined that the target object is in a dumping behavior state.

[0027] The barycentric position information of the target object in each to-be-detected image is obtained according to the plurality of human key points of each to-be-detected image.

[0028] acquire a part key point according to each of the human body key points of the to-be-detected image;

[0029] determine whether a target object in the to-be-detected image is in a preset behavior state based on the part key point;

[0030] if yes, mark the to-be-detected image as a behavior prediction image;

[0031] acquire a barycentric position information of the target object in each of the behavior prediction images according to a part key point of each of the behavior prediction images.

[0032] The part key point comprises an hip key point.

[0033] The behavior recognition method further comprises:

[0034] acquire an hip key point and a human body detection frame of each of the to-be-detected images;

[0035] acquire an upper body proportion of the target object based on the hip key point and the human body detection frame;

[0036] when the upper body proportion is less than a second preset threshold, mark the to-be-detected image as a behavior prediction image.

[0037] The part key point comprises an ankle key point and a knee key point.

[0038] The behavior recognition method further comprises:

[0039] acquire an hip key point and a knee key point of each of the to-be-detected images;

[0040] acquire a calf vector direction based on the hip key point and the knee key point;

[0041] when the calf vector direction is not downward, mark the to-be-detected image as a behavior prediction image.

[0042] The part key point comprises a head key point and an hip key point.

[0043] The behavior recognition method further comprises:

[0044] acquire a head key point and an hip key point of each of the to-be-detected images;

[0045] acquire an upper body direction coordinate based on the head key point and the hip key point;

[0046] acquire a body angle bias according to the upper body direction coordinate;

[0047] When the body angle is greater than a third preset threshold, the to-be-detected image is marked as a behavior prediction image.

[0048] The part key points include a shoulder key point and a hip key point.

[0049] The center of gravity position information of the target object in each behavior prediction image is obtained according to the part key points of each behavior prediction image.

[0050] The coordinate mean value of the shoulder key point and the hip key point of each behavior prediction image.

[0051] The center of gravity position information of the target object is obtained based on the coordinate mean value.

[0052] The to-be-detected images are obtained, including:

[0053] The target object is captured by the accompanying robot to obtain the to-be-detected images.

[0054] The application further provides a behavior recognition device, which comprises a processor and a memory, the memory stores program data, and the processor is used to execute the program data to realize the key point detection method and / or the behavior recognition method.

[0055] The application further provides a computer readable storage medium for storing program data, which is used to realize the key point detection method and / or the behavior recognition method when executed by a processor.

[0056] The application has the beneficial effect that the behavior recognition device uses a plurality of stacked re-parameterized convolution blocks to extract output feature maps of different down-sampling resolutions of the to-be-detected image; the output feature maps of different down-sampling resolutions are up-sampled, and the output feature maps of the same resolution after up-sampling are fused until the final fused feature map is obtained; the coordinate points with pixel values satisfying a preset condition are obtained from the fused feature map as the key points in the to-be-detected image. Through the above manner, the behavior recognition device uses the structure re-parameterization idea to construct a key point detection network composed of stacked re-parameterized convolution blocks, improves the precision and performance of the key point detection network, reduces the parameter amount and calculation amount of the key point detection network, and accelerates the inference speed of the key point detection network. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative work based on these drawings are within the scope of protection of the present application.

[0058] Figure 1 is a flowchart of an embodiment of the key point detection method provided by the present application;

[0059] Figure 2 is a structural diagram of the key point detection network provided by the present application;

[0060] Figure 3 is a network training stage and a network inference stage of the key point detection network provided by the present application;

[0061] Figure 4 is a flowchart of an embodiment of the behavior recognition method provided by the present application;

[0062] Figure 5 is a whole flowchart of the behavior recognition method provided by the present application;

[0063] Figure 6 is a diagram of the human key point distribution provided by the present application;

[0064] Figure 7 is a flowchart of the judgment process of the behavior recognition method provided by the present application;

[0065] Figure 8 is a structural diagram of an embodiment of the behavior recognition device provided by the present application;

[0066] Figure 9 is a structural diagram of an embodiment of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0068] Specifically, please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of an embodiment of the key point detection method provided by the present application, Figure 2 is a structural diagram of the key point detection network provided by the present application.

[0069] The key point detection method of this application is applied to a behavior recognition device, which can be a server or terminal device, or a system in which the server and terminal device cooperate with each other. Accordingly, the various parts of the behavior recognition device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or they can be set in the server and the terminal device respectively.

[0070] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, such as a distributed system or a cloud platform, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software program or software module; no specific limitations are made here. In some possible implementations, the key point detection method in the embodiments of this application can be implemented by the processor calling computer-readable instructions stored in memory.

[0071] Among them, such as Figure 2 As shown, the behavior recognition device first constructs a multi-path key point detection training network. This training network is based on the Simple Baseline framework and the RepVGG network, and utilizes the concept of structural reparameterization to construct and train a network such as... Figure 2 The Simple-Rep network structure is shown. In this embodiment, the keypoint detection training network mainly consists of a backbone network and deconvolution modules. The backbone network mainly consists of sequentially stacked 3×3 convolutional blocks and several reparameterized convolutional blocks, i.e. Figure 2 The RepVGG-Block shown is composed of three deconvolutional layers.

[0072] The Simple-Rep backbone network module is an improvement and optimization based on RepVGG, establishing the RepVGG-small network structure, which consists of four parts: Stage 1, Stage 2, Stage 3, and Stage 4. Compared to the RepVGG network, the RepVGG-small network removes its original Stage 1, retaining the latter four parts, thus ensuring the model's deep feature extraction capability and reducing its number of layers from [2,4,14,1] to [2,4,8,2], further reducing the number of model parameters and achieving a balance between performance and speed. The four stages of the RepVGG-small network correspond to 4x, 8x, 16x, and 32x downsampling feature maps F1, F2, F3, and F4, with channel numbers of 32, 64, 128, and 256, respectively.

[0073] The core of the RepVGG-small network structure is the RepVGG-Block structure based on structure reparameterization, which has two structures for training and inference, and the specific structure is as shown in Figure 3 In the training structure, the RepVGG-Block has two different residual structures, the first residual structure includes a 3x3 convolution branch and a batch normalization (BN) layer branch, a 1x1 convolution branch and a batch normalization (BN) layer branch, and the convolution step is 2, which performs downsampling operation. The second residual structure includes three branches, a 3x3 convolution branch and a batch normalization (BN) layer branch, a 1x1 convolution branch and a batch normalization (BN) layer branch, and a separate batch normalization (BN) layer branch.

[0074] During the training process, the key point detection inference network is converted using the reparameterization idea. Specifically, the training network with multi-path structure is converted into a single-path inference network using the reparameterization idea, which facilitates the deployment and acceleration of the model. The core of the fusion strategy is to use Op fusion strategy to convert all network layers into Conv3x3.

[0075] Specifically, the convolution layers and batch normalization (BN) layers in the RepVGG-Block used by the training network are fused by formula 1, and there are three fusion methods, namely Conv3x3+BN layer fusion, Conv1x1+BN layer fusion, and Conv3x3(convolution kernel set to all 1)+BN layer fusion. The specific fusion calculation formula is as follows:

[0076]

[0077] where W i represents the convolution layer parameter before conversion, μ i represents the mean of the BN layer, σ i represents the variance of the BN layer, γ i and β i respectively represent the scale factor and offset factor of the BN layer, W i ′ and b i ′ respectively represent the weight and bias of the convolution after fusion.

[0078] Further, the fused convolutional layer is converted into Conv3x3, that is, the convolution with different specific convolution kernels is converted into the convolution with a 3x3 convolution kernel. Since the entire residual block can include Conv1x1 branch and Identity branch. For the Conv1x1 branch, the entire conversion process is to replace the 1x1 convolution kernel with the 3x3 convolution kernel, that is, the values in the 1x1 convolution kernel are moved to the center point of the 3x3 convolution kernel; for the Identity branch, the branch does not change the values of the input feature map, and a 3x3 convolution kernel is set, and all 9 positions are set to 1, which keeps the original values after multiplication with the input feature map.

[0079] Merging Conv3x3 in the residual branch. That is, the weights W and the bias B of all branches are superimposed to obtain a fused Conv3x3 network layer.

[0080] Saving the fused network model structure and weight parameters to obtain a final single-path reparameterization lightweight key point detection inference network, that is, a key point detection network as shown in Figure 2 .

[0081] Next, the functions and processing logic of each structure of the key point detection network will be introduced in combination with the processing process of the key point detection network:

[0082] Specifically, as shown in Figure 1 , the key point detection method of the embodiment of the application specifically includes the following steps:

[0083] Step S11: extracting output feature maps of different down-sampling resolutions of the image to be detected by using a plurality of reparameterization convolutional blocks stacked in sequence.

[0084] In the embodiment of the application, the behavior recognition device extracts features through the backbone network module, inputs the image picture to be detected to perform 3x3 convolution, and then sequentially passes through four stages of the backbone network RepVGG-small to obtain feature maps F1, F2, F3, and F4 corresponding to 4 times, 8 times, 16 times, and 32 times down-sampling.

[0085] Step S12: up-sampling the output feature maps of different down-sampling resolutions, and fusing the output feature maps of the same resolution after up-sampling until the final fused feature map is obtained.

[0086] In the embodiment of the present application, the behavior recognition device inputs the obtained feature map F4 into a deconvolution layer, performs upsampling and channel alignment operations, then performs element-by-element addition with F3, inputs the feature map P3 obtained after addition into a deconvolution layer, performs upsampling again, performs channel alignment, performs element-by-element addition with F2, obtains P2 and inputs P2 into the next deconvolution layer, performs upsampling, performs channel alignment operations, performs element-by-element addition with F1, and finally takes the obtained feature map P1 as the final fusion feature map.

[0087] Specifically, the deconvolution module of the key point detection network includes three deconvolution layers. The first deconvolution layer performs upsampling on the obtained feature map F4, performs channel alignment, reduces the dimension from 256 channels to 128, then performs element-by-element addition with F3 to obtain P3. The second deconvolution layer performs upsampling on P3, performs channel alignment operations, and performs element-by-element addition with F2 to obtain P2. The third deconvolution layer performs upsampling on P2, performs channel alignment operations, and performs element-by-element addition with F1 to obtain the fusion feature map P1.

[0088] Step S13: Obtain a coordinate point whose pixel value meets a preset condition from the fusion feature map as a key point in the image to be detected.

[0089] In the embodiment of the present application, the behavior recognition device obtains a pixel coordinate point with the maximum pixel value in the fusion feature map as a key point in the image to be detected. Specifically, the behavior recognition device can input the fusion feature map P1 into a heat map decoding module, perform key point decoding, and obtain position information of the key point.

[0090] Specifically, in order to predict the specific position of the joint key point of the target object in the original image coordinate space, after heat map prediction, that is, after obtaining the fusion feature map P1, the resolution needs to be restored and converted back to the coordinate space of the original image. The coordinates output by the heat map in the original image coordinate space are obtained from the heat map output by the key point detection network. This process is called heat map decoding. The most core part of heat map decoding is to obtain the coordinates of the maximum value point of the predicted heat map by formula 2. This coordinate is scaled back to the original picture space to obtain the final task output. The coordinate calculation formula of heat map decoding is as follows:

[0091]

[0092] However, since the heat map is usually smaller than the network input, that is, the image to be detected, directly taking the maximum value point coordinates as the prediction result will have quantization error. Therefore, sub-pixel level operation is adopted. Specifically, a manual shift operation is performed from the highest activation to the second highest activation direction, that is, the predicted coordinates are offset by 1 / 2 pixel to the second largest value point.

[0093] Specifically, since the fused feature map P1 of the input heatmap decoding module is the same size as the 4x downsampled feature map F1, the relationship between the image to be detected and the fused feature map P1 is also a 4x downsampled relationship. For example, assuming the original image spatial resolution is 512x512, deleting every other row and every other column from this image yields a 256x256 image, which is 2x2 = 4x downsampling. Therefore, the action recognition device can calculate the quantization error offset pixel value of 1 / 2 based on the proportional relationship of the image resolution. Then, the action recognition device shifts the key points in the fused feature map P1 by 1 / 2 pixel towards the direction of the second largest value point and maps the position information of the shifted key points back to the coordinates of the original image space, thereby obtaining the key points of the image to be detected.

[0094] In this embodiment, the behavior recognition device extracts output feature maps of different downsampling resolutions from the image to be detected using a series of sequentially stacked reparameterized convolutional blocks; it then upsamples the output feature maps with different downsampling resolutions and fuses the upsampled output feature maps of the same resolution until a final fused feature map is obtained; coordinate points whose pixel values ​​meet preset conditions are obtained from the fused feature map and used as key points in the image to be detected. Through this method, the behavior recognition device utilizes the concept of structural reparameterization to construct a key point detection network composed of stacked reparameterized convolutional blocks, improving the accuracy and performance of the key point detection network, reducing the number of parameters and computational load, and accelerating the inference speed of the key point detection network.

[0095] based on Figure 1 The keypoint detection method shown in this application also provides an action recognition method, which can be found in the following embodiments. Figure 4 and Figure 5 , Figure 4 This is a flowchart illustrating an embodiment of the behavior recognition method provided in this application. Figure 5 This is a schematic diagram of the overall process of the behavior recognition method provided in this application.

[0096] Specifically, such as Figure 4 As shown, the behavior recognition method in this application embodiment specifically includes the following steps:

[0097] Step S21: Acquire several images to be detected.

[0098] In this embodiment, the behavior recognition device collects video data of the target scene through the camera of the companion robot. The companion robot maintains a constant distance from the human body and follows the human body's movement. The video data collected by the camera is preprocessed, and the video frames are sampled at intervals of 5, that is, one frame is sampled every 5 frames.

[0099] In other embodiments, the behavior recognition device can also utilize indoor cameras, arranged in a way that provides comprehensive surveillance, to acquire real-time video data of indoor target objects.

[0100] like Figure 5 As shown, after acquiring video data, such as video frames, data preprocessing is required. Specifically, the behavior recognition device inputs the video frames obtained from interval sampling into the human target detection module to perform human detection and obtain the detection results. Where N is the number of human figures detected in the current frame. and Let f represent the coordinates of the top-left and bottom-right corners of the i-th bounding box, respectively. i and c i The distribution represents the confidence and category of the i-th detected target.

[0101] In step S21, the several images to be detected are the human detection results of the same target. The behavior recognition device crops the video frames containing human figures according to the human detection boxes to obtain human figure images, and performs image filling to keep each image to be detected the same size.

[0102] Step S22: Obtain several human key points of the target object in each image to be detected using the key point detection method.

[0103] In this embodiment, the human figure detection results are input into a lightweight keypoint detection network to obtain the coordinates of 17 key points on the human body. The keypoint detection network and its keypoint detection method are described below. Figure 1 and Figure 2 The specific implementation methods have been described in detail and will not be repeated here.

[0104] Furthermore, based on the key point detection results, the behavior recognition device obtains the coordinates of 17 key points on the human body. Specifically, the 17 key points on the human body are as follows: Figure 6 As shown, the coordinates of each key point are (x, y). The behavior recognition device inputs the obtained key point coordinates of the human body into the post-processing module to determine whether there is a person tilting behavior. The specific judgment process is as follows: Figure 7 As shown:

[0105] The behavior recognition device calculates the positions of key points on the head and hips. The coordinates of the head key points are determined by the positional information of key points 1, 2, 3, 4, and 5, which are calculated by averaging the coordinate values ​​of these five key points. The coordinates of the hip key points are determined by the positional information of key points 12 and 13, which are calculated by averaging the coordinate values ​​of key points 12 and 13.

[0106] First, the behavior recognition device calculates the upper body ratio. The upper body ratio is calculated according to the hip key point position information and the human body detection frame information. The height information in the hip key point position information is subtracted from the starting height of the human body detection frame, and then divided by the height of the human body detection frame, to obtain the upper body ratio. The upper body ratio is compared with a threshold value, wherein the threshold value is 0.33, which is obtained by statistically analyzing a plurality of personnel dumping video data. In other embodiments, other numerical threshold values can also be set, which are not listed here. When the upper body ratio is less than the set threshold value, the operation of calculating the leg direction is performed, otherwise the key point detection model is returned to predict the key points of the next frame.

[0107] Second, the behavior recognition device calculates the leg direction. When the left ankle key point and the left knee key point are both valid, the left knee key point position information is subtracted from the left ankle key point position information to obtain the left lower leg vector direction. When the right ankle key point and the right knee key point are both valid, the right knee key point position information is subtracted from the right ankle key point position information to obtain the right lower leg vector direction. When the leg direction is not downward, the operation of calculating the body angle bias is performed, otherwise the key point detection model is returned to predict the key points of the next frame.

[0108] Third, the behavior recognition device calculates the body angle bias. When the head key point and the hip key point are both valid, the upper body direction is calculated by subtracting the hip key point position information from the head key point position information to obtain the upper body direction. The body angle bias is calculated according to the coordinate values of the upper body direction. The horizontal coordinate of the upper body direction is divided by the vertical coordinate, and the ratio is the body angle bias. When the body angle bias is greater than the set threshold value, the key point detection result of this frame is stored in the state list, otherwise the key point detection model is returned to predict the key points of the next frame.

[0109] It should be noted that the behavior recognition device can also use one or more of the above first judgment logic, second judgment logic, and third judgment logic, as well as the above judgment order or any other judgment order, to store the detected images that meet the preset behavior state as behavior prediction images in the state list.

[0110] Step S23: Obtain the center of gravity position information of the target object in each detected image according to the plurality of human body key points of each detected image.

[0111] Step S24: Obtain the center of gravity descent speed of the target object by using the center of gravity position information of the target object in the plurality of detected images.

[0112] Step S25: When the center of gravity descent speed is greater than a first preset threshold value, it is determined that the target object is in a dumping behavior state.

[0113] In the embodiment of the present application, the behavior recognition device calculates the gravity center falling speed. When the shoulder key point and the hip key point are effective, the gravity center position is calculated, the average of the position information of the 6 and 7 key points is obtained to obtain the position information of the shoulder key point, and the average of the position information of the shoulder key point and the position information of the hip key point is obtained to obtain the position information of the gravity center.

[0114] The behavior recognition device reads the state list, calculates the gravity center position information of the first and last frames of the state list, subtracts the two gravity center position information, divides by the length of the state list, and obtains the gravity center falling speed. When the gravity center falling speed is greater than the set threshold value, it is judged that the person is dumping, otherwise the next state list is obtained, and the new gravity center falling speed is calculated.

[0115] In the embodiment of the present application, the present application proposes a person dumping behavior recognition method based on a companion robot. The method uses the camera of the companion robot to collect data, uses a human detection model and a reparameterized lightweight key point detection model to detect human key point position information, wherein the key point detection model is based on a Simple Baseline framework and a RepVGG network, uses a structure reparameterization idea to construct a Simple-Rep network structure, improves the accuracy and performance of the model, reduces the parameter amount and calculation amount of the model, and speeds up the model inference speed. Then, a key point detection post-processing module is proposed, which effectively ensures the accuracy of the dumping behavior recognition and reduces the false detection rate.

[0116] The present application proposes a person dumping behavior recognition and discrimination method based on multi-frame key point detection results. The method first calculates the upper body proportion, leg direction and body angle deviation of the human body according to the human key point position information, and then further judges the person dumping according to the gravity center falling speed between multiple frames, effectively avoiding the judgment of activities such as exercise, bending down to pick up things, and going to bed for rest in daily life as person dumping, greatly reducing the false detection rate of person dumping behavior recognition.

[0117] The present application proposes a data collection method based on a companion robot. The method uses the camera of the companion robot to collect data, effectively utilizes the characteristics of the companion robot following the human body, improves the quality of the data, effectively avoids the omission of personnel, and improves the accuracy of the person dumping behavior recognition.

[0118] Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0119] To implement the key point detection method and / or the behavior recognition method of the above-mentioned embodiments, the present application further proposes a behavior recognition device. For details, please refer to Figure 8, Figure 8 FIG. 1 is a structural schematic diagram of an embodiment of a behavior recognition apparatus provided by the present application.

[0120] The behavior recognition apparatus 500 provided by the embodiment of the present application comprises a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled.

[0121] The memory 51 is configured to store program data, and the processor 52 is configured to execute the program data to implement the key point detection method and / or the behavior recognition method described in the above embodiments.

[0122] In the embodiment, the processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 can be an integrated circuit chip with processing capability. The processor 52 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor 52 can also be any conventional processor.

[0123] To implement the key point detection method and / or the behavior recognition method described in the above embodiments, the present application further provides a computer readable storage medium, such as a storage disk or a compact disk. Figure 9 As shown in FIG. 6, the computer readable storage medium 600 is configured to store program data 61, and the program data 61 is used to implement the key point detection method and / or the behavior recognition method described in the above embodiments when executed by a processor.

[0124] The present application further provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is operable to cause a computer to execute the key point detection method and / or the behavior recognition method described in the embodiments of the present application. The computer program product can be a software installation package.

[0125] The key point detection method and / or the behavior recognition method described in the above embodiments of the present application can exist in the form of a software functional unit and be sold or used as an independent product when implemented. Based on this understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (such as a personal computer, a server, or a network device) or a processor (or multiple processors) to perform all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disk, and various other media that can store program codes.

[0126] The above description is merely an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, based on the content of the specification and drawings of the present application, are also included in the patent protection scope of the present application.

Claims

1. A behavior recognition method, characterized by, The behavior recognition method comprises: acquiring a plurality of to-be-detected images; extracting output feature maps of different down-sampling resolutions of the to-be-detected images by using a plurality of re-parameterized convolution blocks stacked in sequence; performing up-sampling on the output feature maps of different down-sampling resolutions, and fusing the output feature maps of the same resolution after up-sampling until a final fused feature map is obtained; acquiring coordinate points with pixel values satisfying a preset condition from the fused feature map as key points in the to-be-detected image; acquiring a plurality of human key points of a target object in each to-be-detected image; acquiring the barycentric position information of the target object in each to-be-detected image according to the plurality of human key points of each to-be-detected image; acquiring the barycentric descent speed of the target object by using the barycentric position information of the target object in a plurality of to-be-detected images; when the barycentric descent speed is greater than a first preset threshold, determining that the target object is in a dumping behavior state; the method of acquiring the barycentric position information of the target object in each to-be-detected image according to the plurality of human key points of each to-be-detected image comprises: acquiring part key points according to the plurality of human key points of each to-be-detected image; judging whether the target object in the to-be-detected image is in a preset behavior state based on the part key points; if yes, marking the to-be-detected image as a behavior prediction image; acquiring the barycentric position information of the target object in each behavior prediction image according to the part key points of each behavior prediction image; the part key points comprise hip key points; the behavior recognition method further comprises: acquiring hip key points and a human body detection frame of each to-be-detected image; acquiring an upper body proportion of the target object based on the hip key points and the human body detection frame; when the upper body proportion is less than a second preset threshold, marking the to-be-detected image as a behavior prediction image.

2. The behavior recognition method according to claim 1, wherein the output feature maps of different down-sampling resolutions comprise a first output feature map, a second output feature map, a third output feature map and a fourth output feature map in descending order of resolution; the method of performing up-sampling on the output feature maps of different down-sampling resolutions, and fusing the output feature maps of the same resolution after up-sampling until a final fused feature map is obtained comprises: performing up-sampling on the fourth output feature map, and fusing the fourth output feature map after up-sampling with the third output feature map to obtain a fifth output feature map; performing up-sampling on the fifth output feature map, and fusing the fifth output feature map after up-sampling with the second output feature map to obtain a sixth output feature map; performing up-sampling on the sixth output feature map, and fusing the sixth output feature map after up-sampling with the first output feature map to obtain the fused feature map.

3. The behavior recognition method according to claim 1 or 2, wherein the method of acquiring coordinate points with pixel values satisfying a preset condition from the fused feature map as key points in the to-be-detected image comprises: Obtain a coordinate point of a maximum pixel value from the fusion feature map as a key point in the to-be-detected image.

4. The behavior recognition method according to claim 3, wherein the obtaining the coordinate point of the maximum pixel value from the fusion feature map as the key point in the to-be-detected image comprises: obtaining an offset pixel value of a quantization error based on a down-sampling relationship between the fusion feature map and the to-be-detected image; offsetting the coordinate point according to the offset pixel value in an activation direction to obtain an offset coordinate point; and mapping the offset coordinate point to an original coordinate space of the to-be-detected image according to the down-sampling relationship to obtain the key point in the to-be-detected image.

5. The behavior recognition method according to claim 4, wherein the activation direction is a direction from the coordinate point of the maximum pixel value to a coordinate point of a second maximum pixel value.

6. The behavior recognition method according to claim 1, wherein the part key points comprise an ankle key point and a knee key point, and the method further comprises: obtaining the ankle key point and the knee key point of each to-be-detected image; obtaining a calf vector direction based on the ankle key point and the knee key point; and labeling the to-be-detected image as a behavior prediction image when the calf vector direction is not downward.

7. The behavior recognition method according to claim 1, wherein the part key points comprise a head key point and a hip key point, and the method further comprises: obtaining the head key point and the hip key point of each to-be-detected image; obtaining an upper body direction coordinate based on the head key point and the hip key point; and obtaining a body angle bias according to the upper body direction coordinate, and labeling the to-be-detected image as a behavior prediction image when the body angle bias is greater than a third preset threshold.

8. The behavior recognition method according to claim 1, wherein the part key points comprise a shoulder key point and a hip key point, and the obtaining the center of gravity position information of the target object in each behavior prediction image based on the part key points of the behavior prediction image comprises: obtaining a coordinate mean value of the shoulder key point and the hip key point of each behavior prediction image; and obtaining the center of gravity position information of the target object based on the coordinate mean value.

9. The behavior recognition method according to claim 1, wherein the obtaining a plurality of to-be-detected images comprises: collecting the plurality of to-be-detected images by a companion robot. The behavior recognition device comprises a processor and a memory, the memory stores program data, and the processor is configured to execute the program data to implement the behavior recognition method according to any one of claims 1 to 9. The computer readable storage medium is configured to store program data, and the program data, when executed by a processor, is configured to implement the behavior recognition method according to any one of claims 1 to 9. ​ ​ ​ ​ ​ ​ ​ ​ 10. A behavior recognition apparatus, characterized by comprising: ​ 11. A computer readable storage medium, characterized in that, ​

Images