A posture prediction method, device and intelligent robot

Abstract

The present application provides a posture prediction method, device and intelligent robot. When performing posture prediction on a target object image, the undetermined posture area included in the target object image is determined based on the posture starting boundary confidence and the posture ending boundary confidence corresponding to each image block of the target object image, and the posture classification of the target object image is detected. If it is determined that the target object image has one or more undetermined posture areas whose fusion confidence reaches the reference confidence of the posture classification, and the target posture classification is consistent with the set posture classification, the target object image is predicted to be the set posture classification. The present application detects the posture area in the target image with the image block as the recognition unit, and predicts the posture classification of the target object image based on the posture area in the target object image and the posture classification determined globally according to the target object image, so that the prediction result is more accurate and reliable.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a posture prediction method, device and intelligent robot. Background Art

[0002] In the process of the continuous development of intelligent robot technology, posture prediction is an important task in the interaction between robots and the environment and humans. However, there are many technical problems in the existing posture prediction technology.

[0003] Current posture prediction technology lacks accuracy in identifying the posture of target objects. In practical applications, robots need to accurately judge the posture of target objects (such as humans, other objects, etc.) in order to make correct responses, but traditional methods are often not accurate enough in determining the starting and ending boundaries of the posture, resulting in unreliable overall posture prediction results. In terms of feature processing, existing technologies fail to fully mine and utilize the representation information of images. The target object images obtained by intelligent robots contain rich information, but traditional posture prediction technology lacks effective means to integrate and deeply mine this information, and cannot fully extract posture-related features, which in turn affects the accuracy of posture prediction. In addition, the existing posture classification prediction lacks an effective guarantee mechanism. Due to the diversity and complexity of postures, the existing classification prediction methods are prone to misclassification, and there are insufficient measures to ensure the accuracy of posture classification in different situations, which affects the effectiveness of subsequent operations of the robot based on the posture prediction results. Summary of the invention

[0004] In view of this, the present application provides a posture prediction method, device and intelligent robot. The technical solution of the present application is implemented as follows:

[0005] On the one hand, the present application provides a posture prediction method, the method comprising: obtaining an image representation matrix of a target object image, the target object image is an object image to be subjected to posture prediction, and the target object image includes one or more image blocks; determining a posture starting boundary confidence and a posture ending boundary confidence corresponding to each image block in the target object image according to the image representation matrix, the posture starting boundary confidence representing the confidence of the image block as the initial position of the posture region, and the posture ending boundary confidence representing the confidence of the image block as the ending position of the posture region; if the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block satisfy the posture Adaptation requirements: determine the area between the first image block and the second image block in the target object image as a pending posture area, and determine the fusion confidence corresponding to the pending posture area based on the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block; predict the posture classification of the target object image based on the image representation matrix to obtain the target posture classification corresponding to the target object image; if it is determined that the target object image has one or more pending posture areas whose fusion confidence reaches the posture classification reference confidence, and the target posture classification is consistent with the set posture classification, predict the posture of the target object image as the set posture classification.

[0006] In a second aspect, the present application provides a posture prediction device, which includes: an image feature extraction module, which is used to obtain an image representation matrix of a target object image, wherein the target object image is an object image to be subjected to posture prediction, and the target object image includes one or more image blocks; a posture region determination module, which is used to determine a posture starting boundary confidence and a posture ending boundary confidence corresponding to each image block in the target object image according to the image representation matrix, wherein the posture starting boundary confidence represents the confidence of the image block as the initial position of the posture region, and the posture ending boundary confidence represents the confidence of the image block as the ending position of the posture region; and a pending region processing module, which is used to determine a posture starting boundary confidence of a first image block and a posture ending boundary confidence of a second image block if the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block are equal. The confidence meets the posture adaptation requirement, the area between the first image block and the second image block in the target object image is determined as the pending posture area, and the fusion confidence corresponding to the pending posture area is determined together with the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block; a target posture determination module is used to predict the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image; a posture classification prediction module is used to predict the posture of the target object image as the set posture classification if it is determined that the target object image has one or more pending posture areas whose fusion confidence reaches the posture classification reference confidence, and the target posture classification is consistent with the set posture classification.

[0007] In a third aspect, the present application provides an intelligent robot comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the program, the steps of the method described in the first aspect above are implemented.

[0008] The beneficial effects of the present application include: when performing posture prediction on a target object image, the embodiments of the present application determine the pending posture region included in the target object image based on the posture start boundary confidence and the posture end boundary confidence corresponding to each image block of the target object image, and detect the posture classification of the target object image. If it is determined that the target object image has one or more pending posture regions whose fusion confidence reaches the posture classification reference confidence, and the target posture classification is consistent with the set posture classification, the target object image is predicted to be the set posture classification. The present application detects the posture region in the target image using the image block as the recognition unit, and predicts the posture classification of the target object image based on the posture region in the target object image and the posture classification determined globally according to the target object image, so that the prediction result is more accurate and reliable.

[0009] In addition, in order to ensure the accuracy of the image representation matrix prediction and prevent the adoption of a large-scale predictor, the prediction accuracy is increased based on distilling a relatively accurate prediction to a small-scale first predictor. An image representation matrix set is obtained, and the image representation matrix set includes multiple image representation matrices and an initial priori posture classification mark corresponding to each image representation matrix, and the initial priori posture classification mark represents the posture classification of the corresponding image representation matrix. Then, a target image representation matrix is ​​determined from the multiple image representation matrices included in the image representation matrix set, and a first visual annotation corresponding to the target image representation matrix is ​​obtained. The first visual annotation represents that the target image representation matrix is ​​the visualization information of the posture classification, so as to have richer features. A binary debugging sample set is generated according to the target image representation matrix and the first visual annotation corresponding to the target image representation matrix, and then the learnable network parameters of the first predictor are optimized according to the binary debugging sample set to obtain a predictor. The first predictor is a pre-debugging predictor, and the number of parameters of the first predictor is less than the preset number, that is, the first predictor is a small and exquisite predictor. The first predictor is optimized based on the binary debugging sample set with richer features, so that the first predictor can learn richer prediction features and improve the prediction performance. In addition, because the predictor is obtained based on a sophisticated small predictor, the predictor has a simpler structure, is less dependent on computing power, saves costs, and has higher universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of the implementation flow of a posture prediction method provided in an embodiment of the present application.

[0011] Figure 2 A schematic diagram of the composition structure of a posture prediction device provided in an embodiment of the present application.

[0012] Figure 3 A schematic diagram of the hardware entity of an intelligent robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The embodiment of the present application provides a posture prediction method, which can be executed by a processor of an intelligent robot. The intelligent robot can refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, a desktop computer, a mobile device, etc. Figure 1 A schematic diagram of the implementation flow of a posture prediction method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0014] Step S100: obtaining an image representation matrix of a target object image, where the target object image is an object image to be subjected to posture prediction, and the target object image includes one or more image blocks.

[0015] In step S100 of the embodiment of the present application, the intelligent robot obtains an image representation matrix of the target object image. The target object image is an object image to be subjected to posture prediction, which contains one or more image blocks. The image representation matrix is ​​a data structure that mathematically represents the target object image, which can comprehensively reflect various feature information of the image. From a technical point of view, the intelligent robot can obtain the matrix in a variety of ways. For example, one feasible way is to use the pixel value information of the image. Assume that the target object image is a simple grayscale image, and each pixel of the image has a grayscale value (the value range is, for example, 0-255). The intelligent robot can regard the image as a two-dimensional matrix, and the rows and columns of the matrix correspond to the horizontal and vertical coordinates of the image, respectively, and the elements in the matrix are the grayscale values ​​of each pixel. But this is only a very basic image representation method, and the actual image representation matrix will contain more complex feature information.

[0016] In more complex situations, in order to obtain the image representation matrix, the intelligent robot may use some specific image feature extraction algorithms. For example, the convolutional layer of the Convolutional Neural Network (CNN) can be used to extract local features of the image. Assuming that the size of the convolution kernel is 3×3 and the step size is 1, when the image is convolved, the convolution kernel slides on the image. Every time it slides to a position, it performs weighted summation and other operations on the local image area (3×3 pixel area) at that position to obtain a new value. These new values ​​constitute the feature map after the convolution operation. As the number of network layers increases, more abstract and representative features will be continuously extracted, and finally an image representation matrix will be formed.

[0017] In addition, the image representation matrix can be obtained by performing various transformations on the image. For example, Fourier transform is performed to convert the image from the spatial domain to the frequency domain, and the coefficients in the frequency domain can also be used as part of the image representation matrix. The formula is: ,in is the function of the original image in the spatial domain, is its representation in the frequency domain, u and v are the frequency domain coordinates, Through this transformation, the characteristics of the image can be described from different angles, thereby enriching the connotation of the image representation matrix.

[0018] Through these technical means, the intelligent robot converts the target object image into an image representation matrix. This matrix will serve as the basic data for subsequent steps, providing data support for operations such as determining the confidence of the starting boundary of the posture, the confidence of the ending boundary of the posture, and predicting the posture classification.

[0019] Step S200: Determine the posture starting boundary confidence and posture ending boundary confidence corresponding to each image block in the target object image based on the image representation matrix, the posture starting boundary confidence represents the confidence of the image block as the initial position of the posture area, and the posture ending boundary confidence represents the confidence of the image block as the ending position of the posture area.

[0020] In step S200 of the embodiment of the present application, the intelligent robot determines the posture starting boundary confidence and posture ending boundary confidence corresponding to each image block in the target object image based on the image representation matrix obtained in step S100. First, the meanings of the two technical terms, posture starting boundary confidence and posture ending boundary confidence, are clarified. The posture starting boundary confidence represents the confidence of the image block as the initial position of the posture area, while the posture ending boundary confidence represents the confidence of the image block as the ending position of the posture area. The confidence here is a measurement indicator, which indicates the degree of determination of the intelligent robot that a certain image block is at a specific posture boundary (start or end), and the value range is usually between 0 and 1. The closer to 1, the more certain the image block is at the corresponding posture boundary position. In order to determine the posture starting boundary confidence and posture ending boundary confidence of each image block, the intelligent robot can adopt a variety of technical means. A feasible method is to use a classification model based on machine learning, such as a support vector machine (Support Vector Machine, SVM). Assume that there is a pre-trained SVM model. The input of this model is some feature vectors in the image representation matrix (feature vectors related to the image block), and the output is the confidence of the starting boundary of the posture or the confidence of the ending boundary of the posture corresponding to the image block. In actual operation, the intelligent robot will extract the feature information related to each image block from the image representation matrix. Take a simple two-dimensional posture prediction as an example, such as predicting the standing posture of a human body. The target object image is a photo containing a person. The image representation matrix may contain feature information such as the person's outline and the position of joint points. For each image block, the intelligent robot selects the outline features, nearby joint point features, etc. corresponding to the image block from the image representation matrix as the input feature vector and inputs it into the SVM model.

[0021] Assume that x i represents the input feature vector corresponding to the i-th image block, and the decision function of the SVM model can be expressed as , where w is the weight vector and b is the bias term. Through this decision function, the intelligent robot can get an output value, and then pass this output value through a mapping function (such as Sigmoid function ,here Map it to a range between 0 and 1 to obtain the pose start boundary confidence or pose end boundary confidence of the image block.

[0022] Another technical approach is to use the neural network model in deep learning. For example, a small neural network is built specifically for calculating the confidence of the posture boundary. The input layer of this neural network receives the feature data related to the image block extracted from the image representation matrix. The middle layer can contain multiple neurons, which process and transform the input data through different weights and activation functions (such as the ReLU function y=max(0, x)). The output layer outputs the confidence of the posture start boundary and the confidence of the posture end boundary.

[0023] For example, for a neural network with an intermediate layer of 10 neurons, the input layer receives a feature vector of length 50 (features related to the image block extracted from the image representation matrix). The dimension of the weight matrix W is 10×50, and the dimension of the bias vector b is 10×1. In the forward propagation process of the neural network, the output h=ReLU(Wx+b) of the intermediate layer is first calculated, and then the output of the intermediate layer passes through an output layer (for example, containing 2 neurons, corresponding to the confidence of the starting boundary of the posture and the confidence of the ending boundary of the posture, respectively), and the final confidence output is obtained through a similar calculation process. The intelligent robot processes each image block in the target object image through such technical means, thereby obtaining the confidence of the starting boundary of the posture and the confidence of the ending boundary of the posture corresponding to each image block. These confidences will play a key role in subsequent steps (such as determining the undetermined posture area, etc.). In addition to the methods mentioned above, the confidence can also be determined by ensemble learning. For example, an ensemble model composed of multiple different types of basic models (such as multiple neural networks with different structures or SVMs with different parameters) is constructed. Each basic model predicts the confidence of the pose start boundary and the pose end boundary of the image block, and then the prediction results of these basic models are fused to obtain the final confidence. The feasible fusion methods include simple averaging method, weighted averaging method, etc.

[0024] Taking the weighted average method as an example, suppose there are three basic models M1, M2, and M3, and their prediction results for the pose starting boundary confidence of a certain image block are p1, p2, and p3 respectively, and the corresponding weights are w1, w2, and w3 respectively (the weights can be set according to the performance evaluation results of the basic models, such as through cross-validation and other methods). Then the final pose starting boundary confidence Through this integrated learning method, the advantages of multiple models can be combined to improve the accuracy and stability of confidence prediction.

[0025] When determining the confidence of the starting boundary and the ending boundary of the posture, the intelligent robot needs to fully consider the diversity and complexity of the target object image. Different types of target object images (such as human images, human images, object images, etc.) have different feature patterns, so when selecting feature extraction methods and model construction, they need to be adjusted according to the specific application scenario. At the same time, in order to improve the accuracy of the confidence, a large amount of training data is required to train the model. These training data should contain target object images in various postures and be annotated with accurate posture starting boundary and posture ending boundary information.

[0026] Step S300: If the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block meet the posture adaptation requirements, the area between the first image block and the second image block in the target object image is determined as the pending posture area, and the fusion confidence corresponding to the pending posture area is determined based on the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block.

[0027] In step S300 of the embodiment of the present application, the intelligent robot determines the undetermined posture area in the target object image under specific conditions and calculates its fusion confidence.

[0028] The undetermined posture region refers to the region in the target object image that the intelligent robot determines may contain the target posture according to certain rules. The fusion confidence is an indicator used to measure the accuracy of the undetermined posture region, which is obtained by combining the confidence of the posture start boundary of the first image block and the confidence of the posture end boundary of the second image block.

[0029] When the intelligent robot performs this step, it needs to first determine whether the confidence of the posture start boundary of the first image block and the confidence of the posture end boundary of the second image block meet the posture adaptation requirements. The posture adaptation requirement is a pre-set rule used to measure the degree of matching between two image blocks in terms of posture start and end. For example, in some posture prediction scenarios, if the confidence of the posture start boundary is higher than a certain threshold (such as 0.6) and the confidence of the posture end boundary is also higher than a specific threshold (such as 0.6), it can be considered that the posture adaptation requirements are met. The setting of this threshold is usually based on a large amount of experimental data and prior knowledge to ensure that the determined image block combination has high reliability in posture prediction. Assume that the target object image is a human motion image, and the image is divided into multiple image blocks. Taking the prediction of the human arm swing posture as an example, an image block may contain the part where the arm starts to swing (the first image block), and its posture start boundary confidence is high, indicating that the intelligent robot is more confident that this image block is the starting part of the arm swing posture; another image block may contain the part where the arm swings to the extreme position and is about to swing back (the second image block), and its posture end boundary confidence is high. If these two confidence levels meet the posture adaptation requirements, the intelligent robot will determine the area between the two image blocks as the pending posture area.

[0030] In order to determine the fusion confidence corresponding to the undetermined posture area, the intelligent robot can use a variety of technical means. A simple method is the weighted average method. Let the confidence of the posture starting boundary of the first image block be p1, and the confidence of the posture ending boundary of the second image block be p2. Assuming that the same weight w=0.5 is given to both (the weight can also be adjusted according to the specific situation, such as the size of the image block, the importance of the position, etc.), then the fusion confidence P=w×p1+w×p2=0.5×p1+0.5×p2.

[0031] Another technical approach is based on a probability model. Assume that the confidence level of the starting boundary p1 and the confidence level of the ending boundary p2 obey specific probability distributions, such as normal distribution. and Intelligent robots can determine the fusion confidence through probability calculation methods such as Bayesian theorem. According to Bayesian theorem, Here, A can be regarded as the event that the undetermined posture area is the correct posture area, and B can be regarded as the event that the posture start boundary confidence p1 of the first image block and the posture end boundary confidence p2 of the second image block are known. By establishing such a probability model and combining it with the prior probability (such as the correct probability of different types of posture areas obtained from a large amount of training data), the intelligent robot can calculate the fusion confidence.

[0032] For another example, when predicting the rotational posture of an object, if the target object image is a two-dimensional projection image of a three-dimensional object, after the image blocks are divided, one image block contains the characteristic part of the object starting to rotate (the first image block), and its posture starting boundary confidence is high, while the other image block contains the characteristic part of the object when the rotation is about to stop (the second image block), and its posture ending boundary confidence is high. After the intelligent robot determines that they meet the posture adaptation requirements, it determines the area between the two as the undetermined posture area. Then calculate the fusion confidence according to the weighted average method or probability model method mentioned above. If the weighted average method is used, and p1=0.7, p2=0.8, according to the equal weight calculation, the fusion confidence P=0.5×0.7+0.5×0.8=0.75.

[0033] In practical applications, determining the undetermined posture area and fusion confidence is of great significance for posture prediction. Different application scenarios may require different methods to determine the posture adaptation requirements and calculate the fusion confidence. For example, in high-precision posture prediction tasks (such as organ posture analysis in medical images), the threshold of posture adaptation requirements may be set higher to ensure accuracy; while in some scenarios with high real-time requirements (such as rapid detection of human posture in video surveillance), a method with less computational effort but relatively lower accuracy may be used to determine the fusion confidence to meet real-time requirements.

[0034] When executing step S300, the intelligent robot needs to comprehensively consider multiple factors such as the characteristics of the target object image, the accuracy requirements of posture prediction, computing resources, etc., and accurately determine the pending posture area and its corresponding fusion confidence by reasonably setting the posture adaptation requirements and adopting the appropriate fusion confidence calculation method, thereby providing strong data support for subsequent posture classification prediction and other operations.

[0035] Step S400: predicting the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image.

[0036] In step S400 of the embodiment of the present application, the intelligent robot predicts the posture classification of the target object image based on the image representation matrix obtained in step S100, thereby obtaining the target posture classification corresponding to the target object image. Posture classification refers to classifying the posture of the target object into a predefined category. For example, in human posture prediction, posture classification may include different categories such as standing, walking, running, sitting, bending, etc.; in object posture prediction, it may be different placement states of the object, such as horizontal placement, vertical placement, tilted placement, etc. These posture classifications are predetermined based on specific application scenarios and research purposes so that the intelligent robot can accurately classify the target object image.

[0037] Intelligent robots can use a variety of technical means to perform posture classification and prediction. Among them, a commonly used technical means is to use a neural network model. For example, Convolutional Neural Network (CNN). CNN has the ability to automatically extract image features and is very suitable for processing image data.

[0038] The intelligent robot provides the image representation matrix as input to the CNN. This image representation matrix contains rich feature information of the target object image. Taking human posture prediction as an example, if the target object image is an image of a person running, the image representation matrix may contain feature information such as the human body contour, the position and angle of the limbs, and the proportion of each part of the body.

[0039] CNN consists of multiple convolutional layers, pooling layers, and fully connected layers. The convolutional layer extracts local features of the image by sliding the convolution kernel on the image representation matrix. For example, a 3x3 convolution kernel slides on the image representation matrix, and each time the weighted sum of the elements in the area covered by the convolution kernel is calculated to generate a new feature map. The pooling layer is used to downsample the feature map to reduce the amount of data while retaining important features. For example, the maximum pooling layer selects the maximum value in a small area (such as a 2x2 area) as the output, which can reduce the data dimension and improve computational efficiency.

[0040] The fully connected layer integrates the features processed by multiple convolutional layers and pooling layers for the final classification prediction. Assuming that a feature vector x is obtained after processing by the previous layer, the neurons in the fully connected layer are linearly combined with this feature vector through the weight w, and then the bias b is added, that is, y=w·x+b, where y is the output of the fully connected layer. In posture classification prediction, the output of the fully connected layer may be a vector, and each element of the vector corresponds to a score of a posture classification. For example, if there are 5 posture classifications (standing, walking, running, sitting, bending over), then the output of the fully connected layer may be a 5-dimensional vector [s1, s2, s3, s4, s5], where si represents the score of the target object image belonging to the i-th posture classification.

[0041] The intelligent robot can determine the target posture classification based on this score vector. One feasible method is to select the category with the highest score as the target posture classification. For example, if s3 is the maximum value in the vector [s1, s2, s3, s4, s5], then the intelligent robot determines the target posture classification of the target object image as running. In addition to CNN, intelligent robots can also use other types of neural networks, such as Recurrent Neural Network (RNN) and its variants, such as Long-Short Term Memory (LSTM) or Gated Recurrent Unit (GRU). In addition, Support Vector Machine (SVM) can also be used for posture classification prediction. The intelligent robot extracts feature vectors from the image representation matrix and then inputs these feature vectors into the SVM model. The goal of SVM is to find a hyperplane that can separate samples of different posture classifications to the greatest extent. Assuming there are two types of training samples for posture classification (Class A and Class B), SVM optimizes the objective function , while satisfying the constraints (where y i is the sample x i The optimal hyperplane is found by using the class label of the hyperplane, w is the normal vector of the hyperplane, and b is the bias. For multi-class posture classification problems, strategies such as One-vs-Rest or One-vs-One can be used to convert them into multiple two-class classification problems for solution.

[0042] In practical applications, in order to improve the accuracy of posture classification prediction, intelligent robots also need to perform model training. For example, when using CNN for training, a large number of target object image samples with posture classification labels are required. During the training process, the parameters of the neural network (such as the weight of the convolution kernel, the weight of the fully connected layer, etc.) are adjusted by minimizing the loss function between the predicted results and the true labels. A feasible loss function is the cross-entropy loss function (Cross-EntropyLoss). For a multi-class classification problem, assume that y is the one-hot encoded vector of the true label (for example, if the target posture classification is running, in the five posture classifications, the position corresponding to running is 1, and the other positions are 0). is the probability distribution vector predicted by the model (such as converting the output of the fully connected layer into a probability distribution through the softmax function), and the cross entropy loss function The intelligent robot back-propagates the gradient of the loss function to each layer of the network through the back-propagation algorithm, thereby updating the parameters, so that the model is continuously optimized during the training process and the accuracy of posture classification prediction is improved.

[0043] When executing step S400, the intelligent robot can accurately predict the posture classification of the target object image based on the image representation matrix by selecting a suitable classification model, such as CNN, RNN or SVM, and performing effective model training, and obtain the target posture classification. This result will be combined with the relevant information of the pending posture area in the subsequent posture prediction process to finally determine the posture of the target object image.

[0044] Step S500: If it is determined that the target object image has one or more undetermined posture regions whose fusion confidence reaches the reference confidence of the posture classification, and the target posture classification is consistent with the set posture classification, the posture of the target object image is predicted to be the set posture classification.

[0045] In step S500 of the embodiment of the present application, the intelligent robot finally predicts the posture of the target object image according to specific conditions. The fusion confidence is an indicator determined in step S300 for measuring the accuracy of the posture area to be determined, which combines the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block. The posture classification reference confidence is a pre-set threshold value used to determine whether the fusion confidence of the posture area to be determined is sufficiently reliable. The target posture classification is the result obtained by predicting the posture classification of the target object image based on the image representation matrix in step S400, and the set posture classification is a known posture category that the target object image is expected to conform to.

[0046] When the intelligent robot performs this step, it needs to check two key conditions. The first is to determine whether the target object image has one or more pending posture areas whose fusion confidence reaches the reference confidence of posture classification. The setting of the reference confidence of posture classification is based on many factors. For example, in an industrial part posture detection scenario with extremely high accuracy requirements, the reference confidence of posture classification may be set higher, such as above 0.8, to ensure that only highly reliable pending posture areas are considered. In some relatively loose scenarios, such as the general posture recognition of human body in ordinary video surveillance, this threshold may be set to around 0.6.

[0047] For example, in human posture prediction, if the target object image is a picture of a person in motion, the intelligent robot has determined multiple undetermined posture regions in step S300 and calculated their respective fusion confidences. Assuming that one of the undetermined posture regions is about the arm swing posture, its fusion confidence is 0.75, and the posture classification reference confidence is set to 0.7, then this undetermined posture region meets the condition that the fusion confidence reaches the posture classification reference confidence.

[0048] The second is to check whether the target posture classification is consistent with the set posture classification. The set posture classification is the expected posture category determined based on the application scenario or prior knowledge. Continuing with the human posture prediction as an example, if the set posture classification is "walking", and the target posture classification predicted by the intelligent robot in step S400 is also "walking", then this condition is also met.

[0049] When these two conditions are met at the same time, the intelligent robot predicts the posture of the target object image as the set posture classification. The logic behind this decision is that the undetermined posture region with the required fusion confidence level indicates that there is a posture region that meets the requirements in the target object image. At the same time, the consistency between the target posture classification and the set posture classification means that it also meets expectations from the perspective of overall posture classification. The combination of the two provides sufficient basis for determining the posture of the target object image.

[0050] In order to ensure the accuracy of this step, the intelligent robot needs to ensure the accuracy and reliability of the previous steps in the entire posture prediction process. For example, when obtaining the image representation matrix in step S100, a suitable feature extraction method should be used to ensure that the image representation matrix can accurately reflect the characteristics of the target object image. If the image representation matrix fails to accurately contain the feature information of key parts of the human body (such as joints) in human posture prediction, it may lead to inaccurate determination of the posture area in subsequent steps, thereby affecting the judgment of step S500.

[0051] Similarly, when determining the confidence of the posture start boundary and the confidence of the posture end boundary in step S200, the model or algorithm used must have high accuracy. If a neural network model is used, it is necessary to ensure that the network structure is reasonable, the training data is sufficient and the annotations are accurate, otherwise it may cause errors in the confidence calculation, affecting the determination of the pending posture area and the final posture prediction.

[0052] When determining the pending posture area and fusion confidence in step S300, the setting of the posture adaptation requirements and the selection of the fusion confidence calculation method are also crucial. For example, if the posture adaptation requirements are set too loosely, some inaccurate pending posture areas may be determined, and even if the fusion confidence calculation method is correct, erroneous information may be introduced into step S500.

[0053] When performing posture classification prediction in step S400, the selected classification model (such as convolutional neural network, support vector machine, etc.) must be fully trained and optimized. If the performance of the classification model is poor, it may lead to incorrect prediction of the target posture classification, so that even if the fusion confidence of the undetermined posture area meets the requirements in step S500, the target posture classification is inconsistent with the set posture classification, and thus the posture of the target object image cannot be correctly predicted.

[0054] When the intelligent robot executes step S500, it finally determines the posture of the target object image by comprehensively considering the relationship between the fusion confidence of the undetermined posture area and the reference confidence of the posture classification, as well as the consistency between the target posture classification and the set posture classification. This process integrates the results of multiple previous steps and is a key decision-making step in the entire posture prediction method.

[0055] As an implementation manner, the target object image includes one or more image regions, and each image region includes one or more image blocks; based on this, step S100, obtaining an image representation matrix of the target object image, may include:

[0056] Step S110: obtaining a region block integration representation matrix corresponding to the target object image according to the image region division result and the image block division result of the target object image;

[0057] Step S120: performing representation information mining on the region block integrated representation matrix to obtain an image representation matrix corresponding to the target object image.

[0058] In step S100 of the embodiment of the present application, the intelligent robot obtains the region block integrated representation matrix corresponding to the target object image based on the image region division result and the image block division result of the target object image. The image region division result refers to the result after the intelligent robot divides the target object image into different regions according to a certain rule or algorithm. These regions are meaningful divisions of the image and may be based on the object structure, semantic information, etc. in the image. For example, when performing image region division on an image containing a person and a background, the main body of the person may be taken as one region, and different objects in the background (such as trees, buildings, etc.) may be taken as different regions. The image block division result is the result after dividing the target object image into smaller image blocks. The image block is a unit smaller than the image area. For example, the image can be divided into a large number of image blocks according to a fixed size (such as 16×16 pixels). In order to obtain the region block integrated representation matrix, the intelligent robot can adopt a variety of technical means. One possible way is a method based on feature descriptors. For an image area, the intelligent robot can calculate a specific feature descriptor for each area, such as a shape feature descriptor (such as a Hu moment, for a binary image of a region), ... , its p+q-order central moment ,in and is the centroid coordinate of the region, and the Hu moment is the normalized combination of the central moment, which can describe the shape characteristics of the region), texture feature descriptors (such as grayscale co-occurrence matrix, which describes texture features by calculating the frequency of occurrence of pixel pairs with different grayscale values ​​in the image in a specific direction and distance), etc. For image blocks, similar feature descriptors can also be calculated, such as color feature descriptors (such as color histograms, which count the distribution of different colors in image blocks). These descriptors are then combined according to certain rules to form a regional block integration representation matrix.

[0059] In step S120, the intelligent robot performs representation information mining on the region block integrated representation matrix to obtain an image representation matrix corresponding to the target object image. Representation information mining is a process of deeply analyzing the region block integrated representation matrix to extract more representative and valuable feature information.

[0060] Intelligent robots can use principal component analysis (PCA) technology to mine representation information. The purpose of PCA is to find the main components (principal components) in the data, reduce the dimension of the data by projecting high-dimensional data into a low-dimensional space, and retain the main feature information of the data. For the regional block integration representation matrix X (assuming it is an n×m matrix, n is the number of samples, m is the number of features), first calculate its covariance matrix , and then solve the eigenvalues ​​of the covariance matrix C And the corresponding eigenvector vi. Sort the eigenvectors according to the size of the eigenvalues, and select the eigenvectors corresponding to the first k eigenvalues ​​(k < m) to form the projection matrix P. Finally, project the region block integration representation matrix X into the low-dimensional space spanned by P to obtain the new image representation matrix Y = XP.

[0061] Take a simple target object image as an example, assuming that the target object image is a picture of a cat on the grass. In step S110, the intelligent robot divides the image area, taking the cat's body as an area, the grass as an area, the sky as an area, etc.; and divides the image blocks at the same time, for example, dividing the image into many 16×16 pixel image blocks. For the cat's body area, the intelligent robot calculates its shape feature descriptor (such as using Hu moments to describe the approximate outline shape of the cat's body), and for the grass area, it can calculate its texture feature descriptor (such as using grayscale co-occurrence matrix to describe the texture of the grass), and for each image block, it can calculate its color feature descriptor (such as color histogram), and then combine these descriptors to form a regional block integrated representation matrix.

[0062] In step S120, the intelligent robot uses PCA to mine the representation information of the region block integrated representation matrix. Since the region block integrated representation matrix contains a lot of feature information (such as feature descriptors of shape, texture, color, etc.), this information may be redundant. PCA finds the main components and projects them into a low-dimensional space, thereby obtaining a more concise image representation matrix that can better represent the image features of the target object. For example, if the original region block integrated representation matrix has 100 feature dimensions, it may be reduced to 20 dimensions after PCA processing. This 20-dimensional image representation matrix can retain the main features of the image, and is more convenient for subsequent operations such as determining the confidence of the starting boundary of the posture, the confidence of the ending boundary of the posture, and predicting the posture classification. In addition to PCA, the intelligent robot can also use Linear Discriminant Analysis (LDA) to mine representation information. The goal of LDA is to find a projection direction so that data of different categories are separated as much as possible in the projected space. For posture prediction, assuming there are target object images of different posture categories (such as a standing cat and a lying cat), LDA will integrate the representation matrix according to the regional blocks of the images of different posture categories and find a projection direction so that the images of different posture categories can be better distinguished in this direction. The calculation process involves the intra-class divergence matrix S w and the inter-class scatter matrix S b The projection direction w is calculated by solving the generalized eigenvalue problem Then, the region block integrated representation matrix is ​​projected to this direction to obtain the image representation matrix.

[0063] In addition, the auto-encoder in deep learning can also be used for representation information mining. The auto-encoder is a neural network structure consisting of an encoder and a decoder. The encoder compresses the input region block integrated representation matrix into a low-dimensional representation (encoding), and the decoder restores this low-dimensional representation to a matrix as close to the original input as possible. During the training process, the network parameters are adjusted by minimizing the reconstruction error between the input and output. The low-dimensional representation output by the encoder can finally be used as the image representation matrix. For example, the region block integrated representation matrix is ​​input into an auto-encoder with multiple hidden layers. After multiple trainings of forward propagation and back propagation, the low-dimensional representation output by the encoder can capture the key feature information in the region block integrated representation matrix, thereby forming an effective image representation matrix.

[0064] As an implementation manner, step S110, based on the image region division result and the image block division result of the target object image, obtains the region block integration representation matrix corresponding to the target object image, which may specifically include:

[0065] Step S111: performing image block vectorization representation on the target object image with the image block as the basic unit, and obtaining an image block representation matrix of the first dimension corresponding to the target object image;

[0066] Step S112: performing image region division on the target object image, performing vector representation of the image region division on the target object image using the image region as a basic unit, and obtaining a first image region representation matrix of a second dimension corresponding to the target object image;

[0067] Step S113: mapping the first image region representation matrix of the second dimension into a second image region representation matrix of the second dimension;

[0068] Step S114: Integrate the image block representation matrix and the second image region representation matrix to obtain a region block integrated representation matrix.

[0069] In step S111, the intelligent robot uses the image block as a basic unit to perform image block vectorization on the target object image, thereby obtaining an image block representation matrix of the first dimension corresponding to the target object image. Image block vectorization is a process of converting feature information of an image block into a vector form. In this process, the intelligent robot needs to determine the features used to describe the image block.

[0070] For example, for a color image block, the intelligent robot can use color features, texture features, etc. to describe it. Taking color features as an example, the intelligent robot can use RGB (red, green, blue) color space to represent the color value of each pixel. Assuming that the size of the image block is m×n pixels, then this image block has m×n pixels. For each pixel, its color can be represented by a three-dimensional vector (R, G, B), where R, G, and B represent the values ​​of the red, green, and blue color channels, respectively, and the value range is usually 0-255. The intelligent robot can arrange the color vectors of these pixels in a certain order (such as row priority), thereby obtaining a vector of length 3×m×n, which is the partial vectorized representation of the image block based on color features.

[0071] In addition to color features, texture features can also be used for image block vectorization. Intelligent robots can calculate the gray-level co-occurrence matrix (GLCM) of image blocks to describe texture features. For a grayscale image block (in order to facilitate the calculation of texture features, the color image can be converted to a grayscale image first, the conversion formula is such as Gray=0.299R+0.587G+0.114B), let the quantization level of its grayscale value be L (for example, L=8, that is, the grayscale value of 0-255 is divided into 8 levels), at a distance of d (such as d=1) and a direction of θ (such as , indicating the horizontal direction), the gray-level co-occurrence matrix P(i, j) indicates the number of times a pixel with gray-level value i and a pixel with gray-level value j, distance d and direction θ appear at the same time in the image block. Intelligent robots can extract some statistics from the gray-level co-occurrence matrix as texture features, such as contrast ,energy By combining the values ​​of these texture features with the color feature vectors, we can obtain a more comprehensive vectorized representation of the image block, which constitutes the image block representation matrix of the first dimension.

[0072] In step S112, the intelligent robot divides the target object image into image regions, and uses the image regions as basic units to perform vector representation of the target object image, and obtains a first image region representation matrix of the second dimension corresponding to the target object image. Image region division is to divide the image into different regions based on factors such as semantic information and object structure in the image. For each image region, the intelligent robot also needs to determine the features used to describe the region. Taking the human figure region as an example, the intelligent robot can use shape features to describe it. For example, calculate the contour perimeter C, area A and some contour-based shape descriptors of the human figure region, such as the Hu moment. For a binary human figure region image I(x, y), its p+q order central moment ,in and It is the centroid coordinate of the area, and the Hu moment is the normalized combination of the central moment. The character area can be described by these shape features.

[0073] In step S113, the intelligent robot maps the first image region representation matrix of the second dimension into the second image region representation matrix of the second dimension. The purpose of this mapping process is to make the dimensions of the image region representation matrix and the image block representation matrix consistent for subsequent integration operations.

[0074] One possible mapping technique is to use a linear mapping method. Assume that the first image region representation matrix is ​​A, whose dimension is m×n, and the intelligent robot wants to map it to a second image region representation matrix B with a dimension of m×k (k is the target dimension, which is consistent with the dimension of the image block representation matrix). The intelligent robot can define a mapping matrix M with a dimension of k×n, and achieve mapping through matrix multiplication B=AM. The mapping matrix M can be determined in a variety of ways, such as by dimensionality reduction methods such as principal component analysis (PCA). Assuming that the first image region representation matrix A is analyzed by PCA to obtain its principal component direction matrix P (with a dimension of n×k), then M=P^T can be set to achieve the purpose of mapping the first image region representation matrix to the second image region representation matrix.

[0075] In step S114, the intelligent robot integrates the image block representation matrix and the second image region representation matrix to obtain a region block integrated representation matrix. The purpose of integrating the two matrices is to combine the feature information of the image block and the image region to more comprehensively describe the target object image.

[0076] The intelligent robot can integrate by direct splicing. For example, let the image block representation matrix be C, whose dimension is p×q, and the second image region representation matrix be D, whose dimension is p×r (due to the previous mapping operation, the p in the two dimensions is the same), then the region block integration representation matrix E can be obtained by splicing these two matrices by column, and the dimension of E is p×(q+r). This integration method is simple and direct, and can effectively integrate the feature information of the image block and the image region, providing a comprehensive feature basis for the subsequent representation information mining (step S120) and other steps of posture prediction.

[0077] Through steps S111-S114, feature extraction, representation and integration are performed from the two levels of image block and image region respectively, and a regional block integrated representation matrix is ​​constructed. This matrix plays an important basic role in the entire posture prediction process and provides necessary data support for subsequent more accurate posture analysis.

[0078] As an implementation manner, step S200, determining the posture start boundary confidence and the posture end boundary confidence corresponding to each image block in the target object image according to the image representation matrix, may include:

[0079] Step S210: loading the image representation matrix into the first neural network to obtain a posture start boundary confidence sequence corresponding to the target object image, wherein the posture start boundary confidence sequence includes the posture start boundary confidence of each image block;

[0080] Step S220: loading the image representation matrix into the second neural network to obtain a posture end boundary confidence sequence corresponding to the target object image, wherein the posture end boundary confidence sequence includes the posture end boundary confidence of each image block;

[0081] Step S400, predicting the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image, includes:

[0082] Step S410: Load the image representation matrix into the third neural network to obtain the target posture classification corresponding to the target object image; wherein the first neural network, the second neural network and the third neural network are obtained based on multi-branch joint debugging.

[0083] In the embodiment S210 of the present application, the intelligent robot loads the image representation matrix into the first neural network to obtain a posture starting boundary confidence sequence corresponding to the target object image, wherein the posture starting boundary confidence of each image block is included in the sequence. The first neural network here is a network model specifically used to determine the posture starting boundary confidence.

[0084] A neural network is a computational model that mimics the structure and function of a biological neural network and is composed of a large number of interconnected neurons. In this process, the image representation matrix is ​​fed into the first neural network as input. For example, assuming that the target object image is an image of a moving human body, the image representation matrix contains the characteristic information of each part of the human body, such as joint positions, body contours, etc. The first neural network may be a multi-layer convolutional neural network (CNN). The convolution layer of the CNN performs a convolution operation by sliding the convolution kernel on the image representation matrix to extract local features of the image.

[0085] Assume that the image representation matrix is ​​X, the convolution kernel is K, and the result of the convolution operation Y can be expressed as , where i and j are the coordinates of the output feature map, and m and n are the coordinates of the convolution kernel. By stacking multiple convolutional layers, the network can gradually extract more abstract and advanced features. After a series of convolution, pooling (such as maximum pooling, which selects the maximum value in a small area as the output to reduce the dimension) and fully connected layer operations, the network finally outputs a sequence of posture starting boundary confidences.

[0086] Taking the prediction of human arm motion posture as an example, if the image block corresponds to the area where the arm starts to move, the first neural network outputs a confidence value indicating that the image block is the starting boundary of the posture based on the feature information of the area in the image representation matrix, such as 0.8, which means that the intelligent robot has a high degree of confidence that this image block is the starting position of the arm motion posture. For other image blocks in the image, they are also processed by the first neural network to obtain the corresponding starting boundary confidence of the posture, and these confidence values ​​constitute the starting boundary confidence sequence of the posture.

[0087] In step S220, the intelligent robot loads the image representation matrix into the second neural network to obtain a posture end boundary confidence sequence corresponding to the target object image, wherein the sequence includes the posture end boundary confidence of each image block. The second neural network is similar to the first neural network and is also a network model constructed for a specific task (determining the posture end boundary confidence).

[0088] Taking the prediction of human arm motion posture as an example, when the image representation matrix is ​​input into the second neural network, it analyzes each image block in the image. Assuming that a certain image block contains the area where the arm is about to swing back to the extreme position, the second neural network outputs a posture end boundary confidence value, such as 0.7, based on the feature information of the image block in the image representation matrix, indicating that this image block is more likely to be the end position of the arm motion posture. After all image blocks are processed by the second neural network, the confidence values ​​obtained constitute the posture end boundary confidence sequence.

[0089] In step S410, the intelligent robot loads the image representation matrix into the third neural network to obtain the target posture classification corresponding to the target object image. The third neural network is a network model specifically used for posture classification tasks.

[0090] For example, for a human posture prediction scenario that includes multiple postures (standing, walking, running, etc.), the third neural network receives the image representation matrix as input. The structure of this network is similar to the previous neural network and may also include convolutional layers, pooling layers, and fully connected layers. Taking a simple convolutional neural network as an example, in the convolutional layer, the features in the image representation matrix are extracted through convolution operations, such as the relative position relationship of various parts of the body in different postures.

[0091] Suppose the input of the fully connected layer of the third neural network is z, the weight of the fully connected layer is W, and the bias is b, then the output y of the fully connected layer can be expressed as y=Wz+b. In the case of multi-posture classification, the output of the fully connected layer may be a vector, and each element of the vector corresponds to the score of a posture classification. For example, if there are three posture classifications (standing, walking, running), the output of the fully connected layer may be a three-dimensional vector [s1, s2, s3], where s1, s2, and s3 represent the scores of the target object image belonging to the standing, walking, and running posture classifications, respectively. The intelligent robot will select the category with the highest score as the target posture classification. For example, if s2 is the maximum value, then the intelligent robot determines the target posture classification of the target object image as walking.

[0092] These neural networks (the first, second, and third neural networks) require a large amount of labeled data when they are constructed and trained. For example, when training the first neural network to determine the confidence of the starting boundary of the pose, a large amount of image data containing known starting boundaries of the pose is required for training. During the training process, a loss function can be defined to measure the difference between the network output and the true label (known starting boundary information of the pose). By minimizing this loss function, the back-propagation algorithm is used to adjust the network's weights and other parameters so that the network can accurately output the confidence of the starting boundary of the pose.

[0093] Similarly, the second and third neural networks follow similar principles when training, except that they have different true labels and loss function designs for different tasks (determining the confidence of the end boundary of the posture and posture classification). For example, when training the third neural network for posture classification, if the cross entropy loss function is used, the one-hot encoding vector y of the true label (such as [0, 1, 0] for walking posture) and the probability distribution vector predicted by the model are , cross entropy loss function .

[0094] Moreover, the first, second and third neural networks are obtained based on multi-branch joint debugging. This joint debugging is to enable each network to work together in the entire posture prediction system to improve the overall prediction performance. During the multi-branch joint debugging, debugging is performed based on a preset error metric, which is obtained by weighting the first error metric corresponding to the first neural network, the second error metric corresponding to the second neural network, and the third error metric corresponding to the third neural network based on a preset influence coefficient, and the first error metric and the second error metric are error metrics of the same type. For example, let the first error metric be E1, the second error metric be E2, the third error metric be E3, and the preset influence coefficients be w1, w2, and w3, respectively, then the preset error metric E=w1E1+w2E2+w3E3. In the joint debugging process, by adjusting the structure and parameters of the network, the preset error metric is minimized, thereby ensuring the coordination between the various networks and the accuracy of the overall posture prediction.

[0095] The intelligent robot uses different neural networks to predict the posture starting boundary confidence, posture ending boundary confidence and posture classification through steps S210-S220 and step S410. These steps are interrelated and work together. They are the key links in the posture prediction method and provide important data and basis for accurately predicting the posture of the target object image.

[0096] As an implementation mode, the first neural network, the second neural network and the third neural network are debugged based on a preset error metric during multi-branch joint debugging. The preset error metric is obtained by weighting a first error metric corresponding to the first neural network, a second error metric corresponding to the second neural network and a third error metric corresponding to the third neural network based on a preset influence coefficient. The first error metric and the second error metric are error metrics of the same type.

[0097] In an embodiment of the present application, the first neural network, the second neural network, and the third neural network are debugged based on a preset error metric during multi-branch joint debugging. This debugging method is intended to enable the three neural networks to work together in the entire posture prediction system to achieve more accurate posture prediction results. The error metric is an indicator used to measure the difference between the prediction result of the neural network and the true result. In this embodiment, the first neural network has a corresponding first error metric, the second neural network has a second error metric, and the third neural network has a third error metric. The first error metric and the second error metric are error metrics of the same type, which means that they are of the same type in terms of the way they measure the difference between the network prediction and the true value, for example, both may be mean square error (MSE) or mean absolute error (MAE), etc.

[0098] The preset error metric is obtained by weighting the error metrics of the first, second and third neural networks based on a preset influence coefficient. The preset influence coefficient is a value preset according to the relative importance of each neural network in the entire posture prediction task.

[0099] For example, in the scenario of human posture prediction, the first neural network is used to determine the confidence of the starting boundary of the posture, the second neural network is used to determine the confidence of the ending boundary of the posture, and the third neural network is used for posture classification. Assume that both the first error metric and the second error metric are measured using mean square error (MSE). For the first neural network, let its predicted confidence of the starting boundary of the posture be , the true starting boundary confidence is , then the first error degree , where N is the number of samples. Similarly, for the second neural network, let its predicted confidence of the end boundary of the posture be , the true pose end boundary confidence is , the second error metric .

[0100] For the third neural network, assuming that it uses the cross entropy loss function as the third error metric E3, in the posture classification task, if there are K posture classifications, the one-hot encoding vector of the true label is , the probability distribution vector predicted by the model is ,but .

[0101] The preset influence coefficients are set to w1, w2, and w3 respectively, then the preset error metric E=w1E1+w2E2+w3E3. The determination of these preset influence coefficients is based on the importance of each neural network in the pose prediction. For example, if the pose classification (the task of the third neural network) is considered to be the most important in the entire task, then the value of w3 may be relatively large; if the determination of the confidence of the start and end boundaries of the pose (the tasks of the first and second neural networks) is relatively equally important, then the values ​​of w1 and w2 may be similar.

[0102] During the multi-branch joint debugging process, the intelligent robot will adjust the parameters of the three neural networks according to this preset error metric. For example, the intelligent robot may use an optimization algorithm based on gradient descent, such as stochastic gradient descent (SGD). For each learnable parameter θ in the neural network (such as the convolution kernel weight in the convolutional neural network, etc.), the gradient of θ with respect to the preset error metric E is calculated. To update the parameters, the update formula is ,in is the learning rate, which determines the step size of each parameter update. By continuously adjusting the parameters, the preset error metric E is gradually reduced, thereby improving the collaborative performance of the three neural networks in the posture prediction task.

[0103] The advantage of this multi-branch joint debugging method based on preset error metrics is that it takes into account the different roles of the three neural networks in the entire posture prediction task and the relationship between them. By reasonably setting the preset influence coefficient, the optimization direction of each network can be balanced according to actual needs and task focus. For example, in some application scenarios that require high confidence in the start and end boundaries of the posture, the values ​​of w1 and w2 can be appropriately increased to ensure that the first and second neural networks can predict the boundary confidence more accurately; in other scenarios that pay more attention to the accuracy of posture classification, the value of w3 can be increased so that the third neural network can get more optimization weight in the joint debugging. In this way, the three neural networks can cooperate with each other to improve the accuracy and stability of the entire posture prediction system.

[0104] As an implementation manner, step S300, if the posture start boundary confidence of the first image block and the posture end boundary confidence of the second image block meet the posture adaptation requirement, determining the area between the first image block and the second image block in the target object image as the undetermined posture area may include:

[0105] Step S310: acquiring a first image block whose posture starting boundary confidence reaches a first reference confidence in a posture starting boundary confidence sequence, the first image block being the mth image block in the target object image, m≥1;

[0106] Step S320: acquiring a second image block whose posture end boundary confidence reaches a second reference confidence in the posture end boundary confidence sequence, where the second image block is the nth image block in the target object image, where n≥1;

[0107] Step S330: If m<n, determine the area between the first image block and the second image block as the posture area to be determined.

[0108] In step S310, the intelligent robot obtains the first image block whose confidence of the posture starting boundary reaches the first reference confidence in the posture starting boundary confidence sequence, where the first image block is the mth image block (m≥1) in the target object image. The posture starting boundary confidence sequence is obtained by loading the image representation matrix into the first neural network, wherein each element corresponds to the confidence of an image block in the target object image as the initial position of the posture area. The first reference confidence is a pre-set threshold value, which is used to filter out image blocks with a high probability of being the posture starting boundary. For example, in a human posture prediction scene, if the target object image is a picture of a person doing exercise, the image is divided into multiple image blocks. Assuming that the first reference confidence is set to 0.6, the intelligent robot searches for image blocks with confidences of 0.6 and above in the posture starting boundary confidence sequence. For one of the image blocks that may be the starting position of the arm raising action, the intelligent robot calculates that its posture starting boundary confidence is 0.7, and this image block meets the requirements and can be determined as the first image block. From a technical point of view, the intelligent robot can implement this search process through simple numerical comparison. In the program, a loop can be set to traverse each element in the posture start boundary confidence sequence (corresponding to the confidence of each image block). When the value of the element is greater than or equal to the first reference confidence, the image block corresponding to the element is marked as a potential first image block. In step S320, the intelligent robot obtains the second image block whose posture end boundary confidence reaches the second reference confidence in the posture end boundary confidence sequence. The second image block is the nth image block (n≥1) in the target object image. The posture end boundary confidence sequence is obtained by loading the image representation matrix into the second neural network, in which each element corresponds to an image block as the confidence of the end position of the posture area. The second reference confidence is also a pre-set threshold value used to determine the possible posture end boundary image block. Continuing with the human posture prediction as an example, it is assumed that the second reference confidence is set to 0.65. In the posture end boundary confidence sequence, for an image block that may be the end position of the arm raising action (such as the arm is raised to the highest point and is about to fall), the intelligent robot calculates that its posture end boundary confidence is 0.7, and this image block meets the requirements and is determined as the second image block. The intelligent robot can also use numerical comparison to mark the image blocks that meet the conditions (confidence is greater than or equal to the second reference confidence) as potential second image blocks when traversing the posture end boundary confidence sequence.

[0109] Finally, in step S330, if m < n, the intelligent robot determines the area between the first image block and the second image block as the undetermined posture area. This condition is set based on the logical order of the postures, and usually the posture start boundary should be before the posture end boundary.

[0110] For example, in the example of the human arm raising action mentioned above, if the first image block of the arm starting to raise the action (assuming m=5) and the second image block of the arm being raised to the highest point and about to fall (assuming n=10) are determined, because 5<10, the intelligent robot will determine the area between the two image blocks as the pending posture area, which may contain all relevant image blocks in the entire process of the arm raising from the starting position to the end position. From the perspective of technical implementation, the intelligent robot can determine the area between the two image blocks by indexing. Once the index positions of the first image block and the second image block in the target object image are determined (assuming that the image blocks of the target object image are arranged in a certain order, such as from left to right, from top to bottom), the intelligent robot can determine this pending posture area by traversing all image blocks with indexes from m to n.

[0111] Based on the confidence of the starting and ending boundaries of the posture, the reference confidence threshold is reasonably set, and the possible posture areas are screened out according to the logical order of the posture. In different posture prediction scenarios, the settings of the first reference confidence and the second reference confidence may be different. For example, in the posture detection of industrial parts with high precision, since more accurate posture area determination is required, these two reference confidences may be set higher, such as the first reference confidence is set to 0.8 and the second reference confidence is set to 0.85. In some scenarios with relatively low requirements for posture prediction accuracy, such as approximate human posture recognition in ordinary video surveillance, these two thresholds can be appropriately lowered, such as the first reference confidence is set to 0.5 and the second reference confidence is set to 0.55.

[0112] At the same time, this method of determining the pending posture area also relies on the accuracy of the previous steps (such as accurately obtaining the posture starting boundary confidence and the posture ending boundary confidence in step S200). If there is an error in the calculation process of the posture starting boundary confidence or the posture ending boundary confidence, it may cause the wrong image block to be determined as the first image block or the second image block, thereby affecting the accurate determination of the pending posture area. For example, if the first neural network calculates the posture starting boundary confidence due to insufficient training data or unreasonable network structure, resulting in the confidence of an image block that is not the actual posture starting position being overestimated and exceeding the first reference confidence, then after this wrong image block is determined as the first image block, the subsequently determined pending posture area may be wrong.

[0113] In addition, in practical applications, the complexity and diversity of the target object image will also have an impact on this process. For complex target object images, such as images containing multiple objects with interrelated postures between objects, it is more difficult to determine the confidence of the posture start and end boundaries and the pending posture area. For example, in an image containing a multi-person interactive scene, the postures of different people are intertwined, and the intelligent robot needs to more accurately distinguish the posture start and end boundaries of each individual, which requires the use of more sophisticated algorithms and models in the previous steps (such as the acquisition of the image representation matrix and the calculation of the posture boundary confidence) to ensure that steps S310-S330 can accurately determine the pending posture area.

[0114] When performing posture prediction on a target object image, the embodiment of the present application determines the pending posture region included in the target object image based on the posture start boundary confidence and the posture end boundary confidence corresponding to each image block of the target object image, and detects the posture classification of the target object image. If it is determined that the target object image has one or more pending posture regions whose fusion confidence reaches the posture classification reference confidence, and the target posture classification is consistent with the set posture classification, the target object image is predicted to be the set posture classification. The present application detects the posture region in the target image using the image block as the recognition unit, and predicts the posture classification of the target object image based on the posture region in the target object image and the posture classification determined globally according to the target object image, so that the prediction result is more accurate and reliable.

[0115] As an implementation mode, step S400 predicts the posture classification of the target object image according to the image representation matrix, and obtains the target posture classification corresponding to the target object image based on the predictor, which is debugged based on the following process:

[0116] Step S10: obtaining an image representation matrix set, wherein the image representation matrix set includes a plurality of image representation matrices and an initial a priori posture classification mark corresponding to each image representation matrix, wherein the initial a priori posture classification mark is used to indicate the posture classification of the corresponding image representation matrix;

[0117] Step S20: determining a target image representation matrix from a plurality of image representation matrices included in the image representation matrix set;

[0118] Step S30: obtaining a first visual annotation corresponding to the target image representation matrix, where the first visual annotation indicates that the target image representation matrix is ​​visual information of posture classification;

[0119] Step S40: generating a binary debugging sample set according to the target image representation matrix and the first visual annotation corresponding to the target image representation matrix;

[0120] Step S50: optimizing learnable network parameters of a first predictor according to the binary debugging sample set to obtain a predictor, wherein the first predictor is a pre-debugging predictor, and the number of parameters of the first predictor is less than a preset number.

[0121] In step S10 of the embodiment of the present application, the intelligent robot obtains a set of image representation matrices, which includes multiple image representation matrices and initial a priori posture classification labels corresponding to each image representation matrix. The image representation matrix is ​​the result of feature extraction and characterization of the target object image. It integrates various information of the image. For example, in human body posture prediction, it may include information such as the shape and position relationship of various parts of the human body. The initial prior posture classification label is used to indicate the posture classification of the corresponding image representation matrix. For example, for an image of a human body standing posture, the initial prior posture classification label of the corresponding image representation matrix is ​​"standing". These image representation matrices and corresponding labels are the basic data for predictor debugging. The intelligent robot can obtain this set by processing a large number of target object images with different postures through the previous steps (such as steps S100-S120).

[0122] In step S20, the intelligent robot determines a target image representation matrix from a plurality of image representation matrices included in the image representation matrix set. The purpose of determining the target image representation matrix is ​​to find those image representation matrices that may have difficulty in predicting posture classification or need further optimization. For example, in an image representation matrix set containing a variety of human postures (standing, walking, running, etc.), due to the complexity of certain postures (such as bending posture), the corresponding image representation matrices may be difficult to accurately predict posture classification, and the image representation matrices corresponding to these postures are more likely to be determined as target image representation matrices. The intelligent robot can adopt some strategies to determine the target image representation matrix, such as screening based on factors such as the diversity of posture classification or prediction difficulty.

[0123] In step S30, the intelligent robot obtains the first visual annotation corresponding to the target image representation matrix. The first visual annotation is visual information used to characterize the target image representation matrix as a posture classification. For example, a bounding box, a key point mark, or a heat map can be used to highlight the key area in the image that leads to a specific classification. Taking human posture prediction as an example, if the target posture classification is "walking", the first visual annotation may be to use a bounding box to frame the leg and foot areas on the image (because these areas are key parts in the walking posture), or to mark the joint positions (such as knees, ankles, etc.) with key points, or to use a heat map to indicate the importance of these areas to the walking posture classification (such as the area near the joint point has a high heat, indicating a greater contribution to the classification). The intelligent robot can use a special annotation tool or based on existing annotation data to obtain the first visual annotation. A possible technical means is to use a convolutional neural network (CNN) to extract features, and then convert the extracted features into visual annotations according to a specific algorithm.

[0124] In step S40, the intelligent robot generates a binary debugging sample set based on the target image representation matrix and the first visual annotation corresponding to the target image representation matrix. Each element in the binary debugging sample set is a binary consisting of the target image representation matrix and its corresponding first visual annotation. For example, for a target image representation matrix M and its corresponding first visual annotation L, they form a binary (M, L), and multiple such binary pairs constitute a binary debugging sample set. This set will be used to optimize the predictor because it contains both the feature information of the image (image representation matrix) and the corresponding visual information (first visual annotation), which can provide richer learning samples for the predictor.

[0125] In step S50, the intelligent robot optimizes the learnable network parameters of the first predictor according to the binary debugging sample set, thereby obtaining a predictor. The first predictor is a pre-trained predictor, and the number of its parameters is less than the preset number, that is, a relatively sophisticated small predictor. In the optimization process, the intelligent robot can use a variety of technical means. For example, a gradient descent algorithm is used to minimize the loss function. Assume that the first predictor is a predictor based on a neural network, and its output is a predicted posture classification , the true posture classification is y, and the cross entropy loss function can be used , where i represents the category index of the classification. The intelligent robot calculates the gradient of the loss function on the learnable network parameters (such as weights w and bias b in the neural network) , and then follow the update formula and (in is the learning rate) to update the parameters, and after multiple iterations, until the loss function converges or reaches the preset number of iterations, the optimized predictor is obtained.

[0126] When determining the target image representation matrix in step S20, the intelligent robot may find that the image representation matrices for some special postures (such as the stretching posture) are prone to misclassification in the previous preliminary prediction due to the uniqueness and complexity of the posture, so these image representation matrices are determined as target image representation matrices. For step S30 to obtain the first visual annotation, taking the running posture of a person as an example, the intelligent robot may use key point markings as the first visual annotation. The positions of key parts such as the head, neck, and limb joints are marked on the human body. The combination and positional relationship of these key points can clearly reflect the posture characteristics of the person when running. At the same time, a heat map can also be used to indicate the importance of the areas around these key points to the classification of the running posture. For example, the areas near the limb joints show higher heat in the heat map, indicating that these areas are more critical to determining the classification of the person's running posture.

[0127] When generating a binary debugging sample set in step S40, each target image representation matrix and its corresponding first visual annotation are combined into a binary. For example, a target image representation matrix of a person's running posture and its corresponding first visual annotation containing key point marks and heat maps are combined into a binary and put into a set. The binary in this set provides rich learning information for the predictor, so that the predictor can learn more complex relationships between the image representation matrix and the posture classification, especially the relationship between the key areas and the posture classification reflected by the visual annotation.

[0128] In the debugging process of the entire predictor, each step is interrelated and indispensable. The quality and diversity of the image representation matrix set obtained in step S10 directly affect the effect of subsequent steps. If the data in the image representation matrix set is not rich enough or there is a deviation (for example, there are too many image representation matrices for a certain posture and too few for another posture), it may cause the determination of the target image representation matrix to be inaccurate, thereby affecting the optimization effect of the entire predictor.

[0129] The accuracy of the target image representation matrix determined in step S20 is also critical. If the image representation matrices that really need to be optimized are not accurately screened out, the binary debugging sample set may not contain sufficient effective information, resulting in the predictor being unable to learn key feature relationships. For example, if the image representation matrices corresponding to some complex postures (such as some difficult yoga movements) are ignored in human posture prediction, the predictor may still be unable to accurately classify these postures.

[0130] The accuracy and effectiveness of the first visual annotation obtained in step S30 are equally important. Inaccurate visual annotation may mislead the learning process of the predictor. For example, if the position of the key point mark is wrong or the heat map does not accurately represent the key area in the character posture prediction, then when generating the binary debugging sample set, these erroneous information will be passed to the predictor, causing the predictor to learn the wrong feature relationship, thereby affecting its prediction accuracy for posture classification.

[0131] The binary debugging sample set generated in step S40 is the direct basis for the optimization of the predictor, and its quality directly determines the performance of the predictor. A binary debugging sample set containing rich and accurate information can enable the predictor to better learn the intrinsic connection between the image representation matrix and the posture classification, thereby improving the prediction accuracy. However, if the binary debugging sample set contains noise or erroneous information, it may cause the predictor to overfit or underfit, and fail to achieve the ideal prediction effect.

[0132] Finally, step S50 optimizes the first predictor through the binary debugging sample set to obtain the final predictor. This predictor will play an important role in the subsequent posture prediction process, and its accuracy and reliability depend on the accurate execution of the previous steps. For example, if the learning rate is improperly set during the optimization process (too large may cause the optimization process to be unstable, and too small may cause the convergence speed to be too slow), it may affect the performance of the predictor. Even if the binary debugging sample set generated in the previous step is of high quality, a predictor with good performance cannot be obtained.

[0133] Through the predictor debugging process of steps S10-S50, starting from obtaining a set of image representation matrices, determining the target image representation matrix, obtaining the first visual annotation, generating a set of binary debugging samples, and finally optimizing the first predictor to obtain the predictor, this process comprehensively considers the feature information and visualization information of the image, and improves the accuracy and reliability of the predictor in posture classification prediction by carefully constructing debugging samples and optimizing the predictor, providing a more accurate classification prediction capability for the entire posture prediction method.

[0134] As an implementation manner, the target image representation matrix is ​​a first type of image representation matrix, and step S20, determining the target image representation matrix from a plurality of image representation matrices included in the image representation matrix set, may include:

[0135] Step S21: predicting a plurality of image representation matrices in the image representation matrix set based on a first predictor, respectively, to obtain a first predicted posture classification of each of the plurality of image representation matrices;

[0136] Step S22: For each image representation matrix, if it is determined that the first predicted posture classification of the image representation matrix is ​​different from the posture classification of the image representation matrix, the image representation matrix is ​​determined as a first type of image representation matrix.

[0137] In step S21 of the embodiment of the present application, the intelligent robot predicts the multiple image representation matrices in the image representation matrix set based on the first predictor, and obtains the first predicted posture classification of each image representation matrix in the multiple image representation matrices. The first predictor here is a predictor that has been processed in advance (such as pre-training), and its function is to classify and predict the posture according to the input image representation matrix.

[0138] The image representation matrix set includes multiple image representation matrices, each of which corresponds to a feature representation of a target object image after processing. For example, in the human posture prediction scenario, these image representation matrices may contain feature information of images of different human postures (such as standing, walking, running, bending, etc.). The first predictor processes each image representation matrix and outputs a predicted posture classification, which is the first predicted posture classification.

[0139] Assume that the first predictor is a neural network-based model, such as a simple multilayer perceptron (MLP). For an image representation matrix X, it is passed through the forward propagation process of MLP to obtain the prediction result. MLP consists of an input layer, a hidden layer, and an output layer. The input layer receives the image representation matrix X. Assume that X is an n-dimensional vector (n depends on the dimension setting of the image representation matrix), and the number of neurons in the input layer is n. The hidden layer linearly transforms the input through the weight matrix w1 and the bias vector b_1, and then obtains the output of the hidden layer through the activation function (such as the ReLU function y=max(0, x)). Assuming that the number of neurons in the hidden layer is m, the output H of the hidden layer can be expressed as H=ReLU(Xw1+b_1), where w1 is an n×m weight matrix and b_1 is an m-dimensional bias vector. The output layer then linearly transforms the hidden layer output H through the weight matrix w2 and the bias vector b_2 to obtain the final prediction result, that is, the first predicted posture classification. , , where w2 is an m×k weight matrix (k is the number of categories for posture classification), is a k-dimensional bias vector.

[0140] Take a simple human posture classification as an example, assuming that there are three posture classifications: standing, walking and running. There are multiple image representation matrices in the image representation matrix set, corresponding to different human posture images. The first predictor predicts each image representation matrix. For example, for an image representation matrix corresponding to a walking posture image, the first predictor may output a classification result representing the walking posture after the above calculation process (e.g., in the output vector, the position corresponding to the walking posture is the maximum value). This result is the first predicted posture classification of the image representation matrix.

[0141] In step S22, for each image representation matrix, if it is determined that the first predicted posture classification of the image representation matrix is ​​different from the posture classification of the image representation matrix, the image representation matrix is ​​determined as the first image representation matrix. The posture classification of the image representation matrix here refers to the original real posture classification of the image, which is known and the correct classification corresponding to the image representation matrix.

[0142] In practical applications, this method of determining the first image representation matrix helps to focus on samples that have a greater impact on improving the performance of the predictor. For example, in a large-scale image pose prediction task, there may be a variety of poses and complex scenes. If all image representation matrices are treated equally, a lot of computing resources may be wasted and it is difficult to effectively improve the accuracy of the predictor. In this way, the intelligent robot can specifically process those image representation matrices that are easily misclassified. For example, in subsequent steps (such as generating debugging samples, etc.), these matrices are used to adjust the parameters of the first predictor so that the predictor can better learn the characteristics of these special cases, thereby improving the overall prediction performance.

[0143] The intelligent robot can determine whether the first predicted posture classification of the image representation matrix is ​​different from the posture classification through a simple comparison operation. For example, the first predicted posture classification and the actual posture classification can be represented as two vectors in the program (such as when the posture classification adopts One-Hot Encoding), and then the two vectors are compared element by element to determine whether there is a difference. If there is a difference, the corresponding image representation matrix is ​​marked as the first image representation matrix for subsequent operations.

[0144] Through steps S21-S22, the intelligent robot uses the first predictor to predict each matrix in the image representation matrix set, and determines the matrix whose predicted classification is different from the actual classification as the first image representation matrix. This process provides a key sample selection basis for subsequent predictor debugging and other operations, which helps to improve the accuracy of the predictor in posture classification prediction.

[0145] As an implementation method, the method may further include:

[0146] Step S1: for each image representation matrix, if it is determined that the first predicted posture classification of the image representation matrix is ​​the same as the posture classification of the image representation matrix, the image representation matrix is ​​determined as a second image representation matrix;

[0147] Step S2: obtaining a second visual annotation corresponding to the second image representation matrix, where the second visual annotation is visual information corresponding to the first predicted posture classification obtained when the first predictor predicts the second image representation matrix to obtain the first predicted posture classification;

[0148] Step S40, generating a binary debugging sample set according to the target image representation matrix and the first visual annotation corresponding to the target image representation matrix, may include:

[0149] Step S41: Generate a binary debugging sample set according to the first image representation matrix and the first visual annotation corresponding to the first image representation matrix, and according to the second image representation matrix and the second visual annotation corresponding to the second image representation matrix.

[0150] In step S1, for each image representation matrix, if the first predicted posture classification of the image representation matrix is ​​determined to be the same as the posture classification of the image representation matrix, the intelligent robot determines the image representation matrix as the second image representation matrix. Several key technical terms are involved here. The first predicted posture classification is the result obtained by predicting the image representation matrix by the first predictor, and the posture classification of the image representation matrix is ​​the real posture classification of the target object image corresponding to the image representation matrix.

[0151] For example, in a human posture prediction scenario, assume that the image representation matrix set contains multiple image representation matrices, each matrix corresponds to a human posture image, and the posture classifications may include standing, walking, running, etc. The first predictor predicts one of the image representation matrices and obtains the first predicted posture classification as "standing", and the real human posture corresponding to this image representation matrix is ​​also "standing", then the intelligent robot will determine this image representation matrix as the second image representation matrix. This operation of determining the second image representation matrix helps to distinguish those image representation matrices that are correctly classified by the first predictor. From a technical point of view, the intelligent robot can achieve this determination process through comparison operations. For example, if the posture classification is represented by One-Hot Encoding, the first predicted posture classification and the real posture classification can both be represented in vector form. Let the first predicted posture classification vector be , the true posture classification vector is y, then by comparing the vector The same is determined by comparing the elements of y and y. If all corresponding elements are equal, it means that the first predicted posture classification is the same as the posture classification, so the image representation matrix is ​​determined to be the second image representation matrix.

[0152] In step S2, the intelligent robot obtains a second visual annotation corresponding to the second image representation matrix, where the second visual annotation is the visual information corresponding to the first predicted posture classification obtained when the first predictor predicts the second image representation matrix to obtain the first predicted posture classification. The visual information can be in a variety of forms, such as bounding boxes, key point markers, or heat maps, which help to intuitively display key areas in the image related to posture classification.

[0153] Taking the standing posture of a human body as an example, if a certain image representation matrix is ​​determined to be the second image representation matrix (that is, the first predictor predicts the same posture classification as the real posture classification as the standing posture), the second visual annotation obtained by the intelligent robot may be to use a bounding box to frame the main outline of the human body on the image (because the overall outline of the human body is an important feature area for judging the standing posture), or to use key points to mark the positions of key joints when the human body is standing (such as ankles, knees, hip joints, shoulder joints, cervical vertebrae and other joints), or to use heat maps to indicate the importance of these areas for the standing posture classification (for example, the area near the joint point has a higher temperature, indicating that these areas are more critical to determining the standing posture classification). From the perspective of technical implementation, the intelligent robot can use similar technical means to obtain the first visual annotation to obtain the second visual annotation. For example, if the first predictor is based on a convolutional neural network (CNN) for prediction, during the prediction process, some layers of the CNN can output feature maps, which can be further converted into visual annotations. Specifically, for a trained CNN, in the process of forward propagation calculation of the second image representation matrix to obtain the first predicted posture classification, intermediate results can be extracted in a specific convolutional layer or fully connected layer, and then converted into second visual annotations in the form of bounding boxes, key point markers or heat maps through a specific algorithm.

[0154] In step S41, the intelligent robot generates a binary debugging sample set based on the first image representation matrix and the first visual annotation corresponding to the first image representation matrix, and based on the second image representation matrix and the second visual annotation corresponding to the second image representation matrix. The binary debugging sample set provides a rich learning sample for the predictor during the debugging process of the predictor.

[0155] For the first type of image representation matrix, since its first predicted pose classification is different from the true pose classification, the two-tuple consisting of these matrices and their corresponding first visual annotations can enable the predictor to learn the previous prediction errors and the relationship between the corresponding image features and the correct classification. For example, in human pose prediction, if the true pose classification of an image representation matrix is ​​"running", but the first predictor predicts "walking", its first visual annotation may highlight some features of leg movement (but there may be misinterpretation), and this image representation matrix and the first visual annotation The two-tuple (M1, L1) (where M1 is the first type of image representation matrix and L1 is the corresponding first visual annotation) is put into the two-tuple debugging sample set.

[0156] The tuples consisting of the second image representation matrix and its corresponding second visual annotation are also important. These tuples allow the predictor to further learn the relationship between the features of the image and the classification when it is correctly classified. For example, an image representation matrix whose true posture is "standing" and is correctly predicted as "standing", and its second visual annotation shows the key contours and joint positions of the human body when standing, the tuple (M2, L2) consisting of this image representation matrix and the second visual annotation (M2, L2) (where M2 is the second image representation matrix and L2 is the corresponding second visual annotation) is also put into the tuple debugging sample set.

[0157] The intelligent robot can construct a set of binary debugging samples by storing these binary groups according to a certain data structure. For example, data structures such as arrays or linked lists can be used. Assuming that an array is used to store the binary debugging sample set, the intelligent robot can store different binary groups (Mi, Li) (i=1, 2, …, where Mi is the first or second image representation matrix and Li is the corresponding visual annotation) into the array in turn to form a binary debugging sample set. This set will serve as the basis for subsequent optimization of the predictor, so that the predictor can learn a more comprehensive relationship between image features and posture classification from these samples, thereby improving the accuracy of the predictor.

[0158] Steps S1-S2 and sub-step S41 in step S40 cooperate with each other to provide more comprehensive and effective learning samples for the optimization of the predictor. By distinguishing the first and second image representation matrices and obtaining their corresponding visual annotations to generate a set of two-tuple debugging samples, the predictor can better adapt to different posture classification situations. Whether it is a case of previous wrong prediction or a correct prediction, key feature information can be learned from it, thereby improving the accuracy and reliability of the entire posture prediction system.

[0159] In actual application scenarios, such as complex motion posture analysis, such as athlete posture analysis in sports events or motion posture monitoring of industrial robots, these steps can ensure that the predictor can accurately identify various postures. For athlete posture analysis, different sports have a variety of complex postures. The binary debugging sample set constructed in this way can enable the predictor to learn the subtle differences between different postures, so as to accurately judge the athlete's posture and provide a reliable data basis for subsequent technical analysis (such as action normative evaluation, sports performance analysis, etc.). In terms of industrial robot motion posture monitoring, accurate posture prediction helps to detect abnormalities in robot movements in a timely manner and ensure the safety and efficiency of the production process.

[0160] As an implementation manner, step S30, obtaining a first visual annotation corresponding to the target image representation matrix, may include:

[0161] Step S31: outputting a first visual annotation based on a second predictor according to the target image representation matrix and the initial priori posture classification mark corresponding to the target image representation matrix, wherein the number of parameters of the second predictor is greater than the number of parameters of the first predictor.

[0162] The target image representation matrix is ​​a mathematical representation of the target object image features after a series of processing, which contains various information related to the target object posture. For example, in the human posture prediction scenario, the target image representation matrix may contain information such as the position, shape and relative relationship of various parts of the human body. The initial prior posture classification label is the posture classification corresponding to the target image representation matrix known in advance, such as "standing", "walking" or "running" posture classification.

[0163] The second predictor is a model for outputting the first visual annotation, and the number of its parameters is greater than the number of parameters of the first predictor. The visual annotation is a way to visually represent information related to pose classification in the image, such as bounding boxes, key point labels, or heat maps.

[0164] When the intelligent robot performs this step, the technical means adopted can be based on the neural network technology in deep learning. Assume that the second predictor is a convolutional neural network (CNN), which has a multi-layer structure, including convolutional layers, pooling layers, and fully connected layers.

[0165] Taking human posture prediction as an example, when the intelligent robot receives the target image representation matrix and the corresponding initial priori posture classification mark (assuming it is a "standing" posture), the target image representation matrix is ​​input into the second predictor. In the convolution layer of CNN, the convolution kernel slides on the target image representation matrix to perform a convolution operation to extract the local features of the image. Let the convolution kernel be K, the target image representation matrix be X, and the calculation formula for the result Y after the convolution operation at a certain position (i, j) is , where m and n are the coordinate ranges of the convolution kernel.

[0166] After multiple convolutional layers, the features of the image are gradually extracted and abstracted. Pooling layers (such as the maximum pooling layer) downsample the output of the convolutional layer, for example, selecting the maximum value in a 2×2 area as the output, which helps reduce the amount of data and retain the main features.

[0167] In the fully connected layer, the features processed by the previous layers are integrated, and the first visual annotation corresponding to the initial prior posture classification mark ("standing" posture) is finally output. If the key point mark is output, the fully connected layer will calculate the coordinate position of each key point in the image based on the learned weights and biases. For example, for the standing posture of the human body, the coordinates of key joints such as the ankle, knee, hip joint, shoulder joint, and cervical spine may be output. If the heat map is output, the fully connected layer will calculate the importance of each area in the image for the "standing" posture classification, expressed as a heat value. The higher the heat, the more important the area is for the judgment of the standing posture.

[0168] Taking a simple object posture prediction as an example, assume that the target object is a tilted rectangular box, and the target image representation matrix contains the shape of the box, edge information, etc. The initial prior posture classification is marked as "tilted placement". The intelligent robot inputs the target image representation matrix into the second predictor (also assumed to be a CNN structure). The convolution layer extracts the edges, corners and other features of the box. After processing by the pooling layer, the fully connected layer outputs the first visual annotation based on these features. If it is a bounding box annotation, the fully connected layer will determine the coordinate information of a rectangular box that can just frame the tilted box; if it is a heat map annotation, it will show the importance of each face, edge and other areas of the box for judging the "tilted placement" posture. For example, the edge area where the box contacts the support surface may show a higher heat in the heat map.

[0169] This process of outputting the first visual annotation based on the second predictor makes full use of the feature information in the target image representation matrix and the prior knowledge of the initial prior pose classification label. The stronger learning ability brought by the larger number of parameters of the second predictor can generate the first visual annotation more accurately, which will provide an important basis for subsequent steps (such as generating a set of binary debugging samples, etc.), and help improve the accuracy of pose prediction.

[0170] As an implementation manner, step S31, outputting a first visual annotation based on a second predictor according to the target image representation matrix and the initial priori posture classification mark corresponding to the target image representation matrix, may include:

[0171] Step S311: generating a first guiding frame according to the target image representation matrix and the initial priori posture classification mark corresponding to the target image representation matrix;

[0172] Step S312: outputting a first visual annotation based on the second predictor according to the first guiding framework.

[0173] In step S311 of the embodiment of the present application, the intelligent robot generates a first guidance frame based on the target image representation matrix and the initial priori posture classification mark corresponding to the target image representation matrix. The target image representation matrix here is a matrix containing rich feature information obtained after processing the target object image. It integrates various information related to the posture in the target object image. For example, in human posture prediction, it may contain information such as the shape, position relationship, and color distribution of various parts of the human body; the initial priori posture classification mark is the posture classification corresponding to the predetermined target image representation matrix. For example, in human posture prediction, it may be posture categories such as "standing", "walking", and "running".

[0174] The first guiding framework is a structure constructed based on the target image representation matrix and the initial prior posture classification mark, which provides a guiding framework for the subsequent output of the first visual annotation based on the second predictor. From a technical point of view, the intelligent robot can generate the first guiding framework in a variety of ways. Taking human posture prediction as an example, it is assumed that the target image representation matrix contains the position information of each joint of the human body and the body contour information, and the initial prior posture classification mark is the "standing" posture. The intelligent robot can build a preliminary framework based on the prior knowledge of the "standing" posture. For example, for the standing posture, the human body's feet should be on the same horizontal line and perpendicular to the ground, the knees should be straight, and the spine should be roughly vertical. The intelligent robot can determine the position of the feet, the degree of knee bending, and the direction of the spine based on the joint position information in the target image representation matrix, and then build a framework containing these key information. This framework may exist in the form of a data structure, such as a vector or matrix containing the position and direction information of each key part.

[0175] In step S312, the intelligent robot outputs the first visual annotation based on the second predictor according to the first guide frame. The second predictor is a model with a large number of parameters, which can output the first visual annotation based on the input information. The first visual annotation here can be in the form of a bounding box, a key point mark or a heat map, etc., which is used to intuitively represent the key areas in the image related to posture classification. Continuing with the "standing" posture in human posture prediction as an example, the first guide frame already contains key information such as the position of both feet, straight knees, and vertical spine. The intelligent robot inputs this first guide frame into the second predictor, and the second predictor (assuming it is a convolutional neural network, CNN) begins to process this input information. In the convolution layer of CNN, the convolution kernel will perform a convolution operation with the information in the first guide frame to extract relevant features. Let the convolution kernel be K, a feature matrix in the first guide frame be F, and the calculation formula for the result G after the convolution operation at a certain position (i, j) is , where m and n are the coordinate ranges of the convolution kernel. After multiple convolutional layers, the information in the first guide frame is gradually extracted and abstracted. The pooling layer of CNN (such as the maximum pooling layer) will downsample the output of the convolution layer, for example, selecting the maximum value in a 2×2 area as the output, which helps to reduce the amount of data and retain the main features. In the fully connected layer, the features processed by the previous layer are integrated, and then the first visual annotation corresponding to the "standing" posture is output according to the pre-trained weights and biases. If it is a key point mark, the fully connected layer will further refine it based on the learned weights and biases on the basis of the first guide frame, and accurately calculate the coordinate position of each key point (such as ankles, knees, hip joints, shoulder joints, cervical vertebrae, etc.) in the image under the human standing posture. If it is a heat map, the fully connected layer will calculate the importance of each area in the image for the "standing" posture classification based on the key information in the first guide frame, and use the heat value to represent it. The higher the heat value, the more important the area is for the judgment of the standing posture.

[0176] This method of outputting the first visual annotation through the second predictor based on the first guidance framework makes full use of the feature information in the target image representation matrix and the prior knowledge provided by the initial prior posture classification mark to construct the first guidance framework. The first guidance framework provides a targeted guidance for the processing of the second predictor, so that the second predictor can output the first visual annotation more accurately. These first visual annotations will play an important role in the subsequent posture prediction process, for example, in operations such as constructing a binary debugging sample set, providing a key basis for improving the accuracy of posture prediction. At the same time, this step-by-step processing method also reflects how to combine prior knowledge with the learning ability of the model in the posture prediction process to improve the accurate representation and prediction ability of posture information.

[0177] As an implementation manner, the image representation matrix set also includes prediction conditions for posture prediction, and each of the multiple image representation matrices corresponds to a posture prediction. Based on this, step S21, based on the first predictor, predicts the multiple image representation matrices in the image representation matrix set respectively to obtain a first predicted posture classification for each of the multiple image representation matrices, which may include:

[0178] Step S211: For each image representation matrix among the multiple image representation matrices, predict the image representation matrix based on a first predictor according to a prediction condition of the posture prediction corresponding to the image representation matrix, and obtain a first predicted posture classification of the image representation matrix under the corresponding posture prediction.

[0179] In the embodiment of the present application, the image representation matrix is ​​a data structure obtained after processing the target object image, which contains a variety of feature information of the target object image, and these feature information are the basis for subsequent posture prediction. For example, in the human body posture prediction scene, the image representation matrix may contain information such as the shape, position relationship, color distribution, etc. of various parts of the human body, and these information can reflect the posture characteristics of the human body when combined. The prediction conditions of posture prediction are specific conditions related to each posture prediction. For example, in different application scenarios, there may be different prediction conditions. In the posture prediction of sports events, if the starting posture of an athlete is predicted, the prediction conditions may be related to the body posture and muscle state at the moment of starting; when predicting the posture of pedestrians in daily monitoring scenarios, the prediction conditions may be related to factors such as the walking direction of pedestrians and the surrounding environment. The first predictor is a pre-built model for posture prediction, which can output the prediction result according to the input image representation matrix and the relevant prediction conditions, that is, the first predicted posture classification. This predictor can be constructed based on a variety of machine learning or deep learning algorithms, such as a convolutional neural network (CNN) or a support vector machine (SVM).

[0180] When the intelligent robot performs this step, it will operate on each image representation matrix. Taking human posture prediction as an example, it is assumed that the image representation matrix set contains multiple image representation matrices corresponding to different human posture images, and each posture prediction has its specific prediction conditions.

[0181] If the first predictor is a convolutional neural network (CNN), the intelligent robot will feed the image representation matrix as input data into the CNN. At the same time, considering the prediction conditions of the posture prediction, the intelligent robot will properly process these conditions so that they can be used by the CNN. For example, if the prediction conditions involve surrounding environmental information (such as whether there are obstacles around pedestrians in the monitoring scenario), the intelligent robot can encode this environmental information and convert it into a form that can be used as CNN input together with the image representation matrix.

[0182] Let the image representation matrix be X, and the information after prediction conditional encoding be C. When inputting them into CNN, they can be combined into a new input vector Z=[X;C] by splicing (where ";" represents the vector splicing operation). Then, the convolution layer of CNN will perform a convolution operation on this new input vector Z. Let the convolution kernel be K. For the result Y(i,j) at a certain position (i,j) of the convolution layer output, the calculation formula is , where m and n are the coordinate ranges of the convolution kernel.

[0183] After multiple convolutional layers and pooling layers (such as the maximum pooling layer, which selects the maximum value in a small area as the output to reduce the amount of data and retain the main features), the fully connected layer integrates the features processed by the previous layers. Finally, the fully connected layer calculates the first predicted posture classification based on the pre-trained weights W and bias b. , the calculation formula is (where f is the input feature vector of the fully connected layer, which is the result of processing by the previous layers).

[0184] This method of performing posture prediction based on the prediction conditions corresponding to the image representation matrix and the first predictor can fully consider various factors related to the posture, thereby improving the accuracy of the first predicted posture classification. Different prediction conditions provide more context information for posture prediction, so that the first predictor can analyze the image representation matrix more comprehensively, and then obtain a prediction result that is more in line with the actual situation.

[0185] As an implementation mode, a plurality of image representation matrices correspond to a plurality of posture predictions. In step S211, according to the prediction condition of the posture prediction corresponding to the image representation matrix, the image representation matrix is ​​predicted based on the first predictor to obtain the first predicted posture classification of the image representation matrix under the corresponding posture prediction. The method may further include:

[0186] Step S211a: for each prediction condition of posture prediction, generate a second guide frame according to the prediction condition;

[0187] Step S211b: According to the second guidance framework, the prediction condition is standardized based on the second predictor to obtain a target prediction condition;

[0188] Step S211, predicting the image representation matrix based on a first predictor according to a prediction condition of a posture prediction corresponding to the image representation matrix, and obtaining a first predicted posture classification of the image representation matrix under the corresponding posture prediction, may include:

[0189] Step S2111: According to the target prediction condition of the posture prediction corresponding to the image representation matrix, the image representation matrix is ​​predicted based on the first predictor to obtain a first predicted posture classification of the image representation matrix under the corresponding posture prediction.

[0190] In step S211a of the embodiment of the present application, the intelligent robot generates a second guidance frame according to the prediction conditions for each posture prediction. The prediction conditions for posture prediction here refer to some limiting factors or prior knowledge related to each specific posture when performing posture prediction. For example, in a human posture prediction scenario, if the starting posture of an athlete is to be predicted, the prediction conditions may include the position of the feet at the start, the forward tilt angle of the body, the swing amplitude of the arms, etc.; if the dancer's dance posture is to be predicted, the prediction conditions may involve the specific rhythm of the dance movements, the relative position relationship of various parts of the body, and the posture characteristics required by the dance style, etc.

[0191] The second guidance framework is a data structure or model framework built based on prediction conditions, which aims to provide a guiding structure for subsequent processing. Taking the prediction of the athlete's starting posture as an example, the intelligent robot constructs the second guidance framework based on the prediction conditions such as the position of the feet at the start (such as standing with both feet forward and backward, and the position of the front foot is a certain distance behind the starting line), the body's forward leaning angle (assuming that the forward leaning angle is within a certain range, such as 10-30 degrees), and the arm's swing amplitude (such as the angle between the arm and the body when the arm swings back to the extreme position, etc.). From a technical point of view, this framework can be represented as a vector or matrix structure containing these key information. For example, the foot position information can be represented as a coordinate vector, the body's forward leaning angle can be represented as a numerical value, and the arm's swing amplitude can be represented as an angle value, and then these information can be combined into a second guidance framework in the form of a vector.

[0192] In step S211b, the intelligent robot standardizes the prediction conditions based on the second predictor according to the second guide frame to obtain the target prediction conditions. The second predictor here is a model that plays a specific role in the posture prediction process. It may have different structures and functions, for example, it can be a model based on a neural network or other machine learning models. The purpose of standardization is to convert the prediction conditions into a form that is more suitable for subsequent processing (such as posture prediction based on the first predictor). Continuing with the athlete's starting posture prediction as an example, assume that the second predictor is a neural network model. The intelligent robot inputs the previously constructed second guide frame (including information such as the position of the feet, the forward leaning angle of the body, the arm swing amplitude, etc.) into the second predictor. Inside the neural network, calculations are performed through neurons in each layer. For example, in the convolution layer, the convolution kernel is convolved with the input frame information. Let the convolution kernel be K, and a feature matrix in the input second guide frame be F. The calculation formula of the result G after the convolution operation at a certain position (i, j) is , where m and n are the coordinate ranges of the convolution kernel.

[0193] After multiple convolutional layers and pooling layers (such as the maximum pooling layer, which selects the maximum value in a small area as the output to reduce the amount of data and retain the main features), the fully connected layer will integrate the features processed by the previous layers and finally output a standardized target prediction condition. This target prediction condition may be an adjusted and optimized numerical representation or feature vector, which better meets the requirements of the entire posture prediction system.

[0194] In step S2111, the intelligent robot predicts the image representation matrix based on the first predictor according to the target prediction condition of the posture prediction corresponding to the image representation matrix, and obtains the first predicted posture classification of the image representation matrix under the corresponding posture prediction. The image representation matrix here is a data structure containing rich feature information obtained after processing the target object image. For example, in human posture prediction, it may contain information such as the shape, position relationship, color distribution, etc. of various parts of the human body.

[0195] The first predictor is a model for posture prediction, which can be built based on different algorithms, such as neural networks, support vector machines, etc. Taking human posture prediction as an example, assume that the first predictor is a convolutional neural network (CNN). The intelligent robot takes the image representation matrix and the corresponding target prediction conditions (conditions related to posture prediction after normalization in the previous steps) as input. For CNN, the input needs to be processed first. Assuming that the image representation matrix is ​​X and the target prediction condition is C, they can be combined into a new input vector Z=[X;C] by splicing (here ";" represents the splicing operation of the vector).

[0196] Then, the convolution layer of CNN will perform convolution operation on this new input vector Z. Let the convolution kernel be K. For the result Y(i,j) at a certain position (i,j) output by the convolution layer, the calculation formula is , where m and n are the coordinate ranges of the convolution kernel.

[0197] After being processed by multiple convolutional layers and pooling layers (such as the maximum pooling layer), the fully connected layer will integrate the features processed by the previous layers. Finally, the fully connected layer calculates the first predicted posture classification based on the pre-trained weights W and bias b. , the calculation formula is (Where f is the input feature vector of the fully connected layer, which is the result of processing by the previous layers.) For example, in the prediction of the athlete's starting posture, if there are different posture classifications such as "standing ready to start", "already started", and "starting and accelerating", through the above calculation, the first predictor will output a result representing the most likely posture classification, such as "already started".

[0198] This process of constructing a second guidance framework, standardizing prediction conditions, and performing posture prediction based on target prediction conditions fully considers various prior knowledge and specific conditions in posture prediction and integrates them into the entire prediction process. This helps improve the accuracy of posture prediction, especially in complex posture prediction scenarios, such as posture prediction under multi-object interaction or complex human motion posture prediction. Different prediction conditions are reasonably integrated and processed in this way, allowing the first predictor to better use this information for accurate posture classification prediction.

[0199] In the training phase of the predictor, in order to ensure the accuracy of the image representation matrix prediction and to prevent the adoption of a large-scale predictor, the embodiment of the present application increases the prediction accuracy by distilling a relatively accurate prediction into a small-scale first predictor. An image representation matrix set is obtained, and the image representation matrix set includes multiple image representation matrices and an initial priori posture classification mark corresponding to each image representation matrix, and the initial priori posture classification mark represents the posture classification of the corresponding image representation matrix. Then, a target image representation matrix is ​​determined from the multiple image representation matrices included in the image representation matrix set, and a first visual annotation corresponding to the target image representation matrix is ​​obtained. The first visual annotation represents that the target image representation matrix is ​​the visualization information of the posture classification, so as to have richer features. A binary debugging sample set is generated based on the target image representation matrix and the first visual annotation corresponding to the target image representation matrix, and then the learnable network parameters of the first predictor are optimized based on the binary debugging sample set to obtain a predictor. The first predictor is a pre-debugging predictor, and the number of parameters of the first predictor is less than the preset number, that is, the first predictor is a sophisticated small predictor. The first predictor is optimized based on a set of binary debugging samples with richer features, so that the first predictor can learn richer prediction features and improve prediction performance. In addition, because the predictor is obtained based on a small and sophisticated predictor, the predictor has a simpler structure, is less dependent on computing power, saves costs, and has higher universality.

[0200] Based on the above embodiments, the present application provides a posture prediction device. Figure 2 A schematic diagram of the structure of a posture prediction device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the posture prediction device 200 includes:

[0201] An image feature extraction module 210 is used to obtain an image representation matrix of a target object image, wherein the target object image is an image of an object to be subjected to posture prediction, and the target object image includes one or more image blocks;

[0202] A posture region determination module 220, configured to determine a posture start boundary confidence and a posture end boundary confidence corresponding to each image block in the target object image according to the image representation matrix, wherein the posture start boundary confidence represents the confidence of the image block as the initial position of the posture region, and the posture end boundary confidence represents the confidence of the image block as the end position of the posture region;

[0203] The pending region processing module 230 is used to determine the region between the first image block and the second image block in the target object image as the pending posture region if the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block meet the posture adaptation requirement, and determine the fusion confidence corresponding to the pending posture region based on the posture starting boundary confidence of the first image block and the posture ending boundary confidence of the second image block;

[0204] A target posture determination module 240 is used to predict the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image;

[0205] The posture classification prediction module 250 is used to predict the posture of the target object image as the set posture classification if it is determined that the target object image has one or more undetermined posture areas whose fusion confidence reaches the posture classification reference confidence, and the target posture classification is consistent with the set posture classification.

[0206] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0207] An embodiment of the present application also provides an intelligent robot, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0208] Figure 3 A schematic diagram of a hardware entity of an intelligent robot provided in an embodiment of the present application, such as Figure 3 As shown, the hardware entity of the intelligent robot 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

Claims

1. A posture prediction method, characterized in that: The method comprises: Acquire an image representation matrix of a target object image, where the target object image is an object image to be subjected to posture prediction, and the target object image includes one or more image blocks; Determine, according to the image representation matrix, a posture start boundary confidence and a posture end boundary confidence corresponding to each image block in the target object image, wherein the posture start boundary confidence represents the confidence of the image block as the initial position of the posture region, and the posture end boundary confidence represents the confidence of the image block as the end position of the posture region; If the posture start boundary confidence of the first image block and the posture end boundary confidence of the second image block meet the posture adaptation requirement, determine the area between the first image block and the second image block in the target object image as a pending posture area, and determine the fusion confidence corresponding to the pending posture area according to the posture start boundary confidence of the first image block and the posture end boundary confidence of the second image block; Predicting the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image; If it is determined that the target object image has one or more undetermined posture regions whose fusion confidence reaches the reference confidence of the posture classification, and the target posture classification is consistent with the set posture classification, predicting the posture of the target object image as the set posture classification; Wherein, determining the posture start boundary confidence and the posture end boundary confidence corresponding to each image block in the target object image according to the image representation matrix includes: Loading the image representation matrix into a first neural network to obtain a posture start boundary confidence sequence corresponding to the target object image, wherein the posture start boundary confidence sequence includes the posture start boundary confidence of each image block; Loading the image representation matrix into a second neural network to obtain a posture end boundary confidence sequence corresponding to the target object image, wherein the posture end boundary confidence sequence includes the posture end boundary confidence of each image block; The step of predicting the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image includes: Loading the image representation matrix into a third neural network to obtain the target posture classification corresponding to the target object image; Wherein, the first neural network, the second neural network and the third neural network are obtained based on multi-branch joint debugging; During the multi-branch joint debugging, the first neural network, the second neural network, and the third neural network are debugged based on a preset error metric, wherein the preset error metric is obtained by weighting a first error metric corresponding to the first neural network, a second error metric corresponding to the second neural network, and a third error metric corresponding to the third neural network based on a preset influence coefficient, and the first error metric and the second error metric are error metrics of the same type; If the posture start boundary confidence of the first image block and the posture end boundary confidence of the second image block meet the posture adaptation requirement, determining the area between the first image block and the second image block in the target object image as the undetermined posture area, comprising: Acquire the first image block whose posture starting boundary confidence reaches a first reference confidence in the posture starting boundary confidence sequence, the first image block being the mth image block in the target object image, m≥1; Acquire the second image block whose posture end boundary confidence reaches a second reference confidence in the posture end boundary confidence sequence, the second image block being the nth image block in the target object image, where n≥1; If m<n, the area between the first image block and the second image block is determined as the undetermined posture area.

2. The method according to claim 1, characterized in that The target object image includes one or more image areas, and each of the image areas includes one or more image blocks; The step of obtaining an image representation matrix of the target object image includes: Obtaining a region block integration representation matrix corresponding to the target object image according to the image region division result and the image block division result of the target object image; The representation information of the integrated representation matrix of the region block is mined to obtain an image representation matrix corresponding to the target object image.

3. The method according to claim 2, characterized in that The step of obtaining a region block integration representation matrix corresponding to the target object image based on the image region division result and the image block division result of the target object image includes: Performing image block vectorization on the target object image with image blocks as basic units to obtain an image block representation matrix of a first dimension corresponding to the target object image; Performing image region division on the target object image, performing vector representation of the image region division on the target object image using the image region as a basic unit, and obtaining a first image region representation matrix of a second dimension corresponding to the target object image; Mapping the first image region representation matrix of the second dimension into a second image region representation matrix of the second dimension; The image block representation matrix and the second image region representation matrix are integrated to obtain the region block integrated representation matrix.

4. The method according to claim 1, characterized in that: The step of predicting the posture classification of the target object image according to the image representation matrix and obtaining the target posture classification corresponding to the target object image is implemented based on a predictor, and the predictor is debugged based on the following process: Acquire an image representation matrix set, wherein the image representation matrix set includes a plurality of image representation matrices and an initial a priori posture classification mark corresponding to each image representation matrix, wherein the initial a priori posture classification mark is used to indicate a posture classification of the corresponding image representation matrix; Determining a target image representation matrix from a plurality of image representation matrices included in the image representation matrix set; Acquire a first visual annotation corresponding to the target image representation matrix, where the first visual annotation indicates that the target image representation matrix is ​​visual information of the posture classification; Generate a binary debugging sample set according to the target image representation matrix and a first visual annotation corresponding to the target image representation matrix; The learnable network parameters of the first predictor are optimized according to the binary debugging sample set to obtain the predictor, wherein the first predictor is a pre-debugging predictor and the number of parameters of the first predictor is less than a preset number.

5. The method according to claim 4, characterized in that The target image representation matrix is ​​a first type of image representation matrix, and determining the target image representation matrix from a plurality of image representation matrices included in the image representation matrix set comprises: Predicting a plurality of image representation matrices in the image representation matrix set respectively based on the first predictor to obtain a first predicted posture classification of each image representation matrix in the plurality of image representation matrices; For each image representation matrix, if it is determined that the first predicted posture classification of the image representation matrix is ​​different from the posture classification of the image representation matrix, then determining the image representation matrix as the first image representation matrix; For each image representation matrix, if it is determined that the first predicted posture classification of the image representation matrix is ​​the same as the posture classification of the image representation matrix, then determining the image representation matrix as a second image representation matrix; Obtaining a second visual annotation corresponding to the second image representation matrix, where the second visual annotation is visual information corresponding to the first predicted posture classification obtained when the first predictor predicts the second image representation matrix to obtain the first predicted posture classification; The step of generating a binary debugging sample set according to the target image representation matrix and the first visual annotation corresponding to the target image representation matrix includes: generating the binary debugging sample set according to the first image representation matrix and the first visual annotation corresponding to the first image representation matrix, and according to the second image representation matrix and the second visual annotation corresponding to the second image representation matrix; The obtaining of a first visual annotation corresponding to the target image representation matrix includes: The first visual annotation is output based on a second predictor according to the target image representation matrix and the initial priori posture classification label corresponding to the target image representation matrix, and the number of parameters of the second predictor is greater than the number of parameters of the first predictor.

6. The method according to claim 5, characterized in that The step of outputting the first visual annotation based on the target image representation matrix and the initial priori posture classification label corresponding to the target image representation matrix and based on the second predictor comprises: generating a first guiding frame according to the target image representation matrix and an initial priori posture classification mark corresponding to the target image representation matrix; outputting the first visual annotation based on the second predictor according to the first guided framework; The image representation matrix set also includes prediction conditions for posture prediction, each of the multiple image representation matrices corresponds to a posture prediction, and the multiple image representation matrices in the image representation matrix set are predicted based on the first predictor to obtain a first predicted posture classification for each of the multiple image representation matrices, including: For each image representation matrix among the multiple image representation matrices, the image representation matrix is ​​predicted based on the first predictor according to the prediction condition of the posture prediction corresponding to the image representation matrix to obtain a first predicted posture classification of the image representation matrix under the corresponding posture prediction.

7. A posture prediction device, characterized in that: The device comprises: An image feature extraction module, used to obtain an image representation matrix of a target object image, wherein the target object image is an image of an object to be subjected to posture prediction, and the target object image includes one or more image blocks; A posture region determination module, used to determine the posture start boundary confidence and the posture end boundary confidence corresponding to each image block in the target object image according to the image representation matrix, wherein the posture start boundary confidence represents the confidence of the image block as the initial position of the posture region, and the posture end boundary confidence represents the confidence of the image block as the end position of the posture region; A pending region processing module, configured to determine a region between the first image block and the second image block in the target object image as a pending posture region if the posture start boundary confidence of the first image block and the posture end boundary confidence of the second image block meet the posture adaptation requirement, and determine a fusion confidence corresponding to the pending posture region based on the posture start boundary confidence of the first image block and the posture end boundary confidence of the second image block; A target posture determination module is used to predict the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image; A posture classification prediction module, configured to predict the posture of the target object image as the set posture classification if it is determined that the target object image has one or more undetermined posture regions whose fusion confidence reaches the reference confidence of the posture classification, and the target posture classification is consistent with the set posture classification; Wherein, determining the posture start boundary confidence and the posture end boundary confidence corresponding to each image block in the target object image according to the image representation matrix includes: Loading the image representation matrix into a first neural network to obtain a posture start boundary confidence sequence corresponding to the target object image, wherein the posture start boundary confidence sequence includes the posture start boundary confidence of each image block; Loading the image representation matrix into a second neural network to obtain a posture end boundary confidence sequence corresponding to the target object image, wherein the posture end boundary confidence sequence includes the posture end boundary confidence of each image block; The step of predicting the posture classification of the target object image according to the image representation matrix to obtain the target posture classification corresponding to the target object image includes: Loading the image representation matrix into a third neural network to obtain the target posture classification corresponding to the target object image; Wherein, the first neural network, the second neural network and the third neural network are obtained based on multi-branch joint debugging; During the multi-branch joint debugging, the first neural network, the second neural network, and the third neural network are debugged based on a preset error metric, wherein the preset error metric is obtained by weighting a first error metric corresponding to the first neural network, a second error metric corresponding to the second neural network, and a third error metric corresponding to the third neural network based on a preset influence coefficient, and the first error metric and the second error metric are error metrics of the same type; If the posture start boundary confidence of the first image block and the posture end boundary confidence of the second image block meet the posture adaptation requirement, determining the area between the first image block and the second image block in the target object image as the undetermined posture area, comprising: Acquire the first image block whose posture starting boundary confidence reaches a first reference confidence in the posture starting boundary confidence sequence, the first image block being the mth image block in the target object image, m≥1; Acquire the second image block whose posture end boundary confidence reaches a second reference confidence in the posture end boundary confidence sequence, the second image block being the nth image block in the target object image, where n≥1; If m<n, the area between the first image block and the second image block is determined as the undetermined posture area.

8. An intelligent robot, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 6 are implemented.

Images