Network training method and apparatus for pose estimation, pose estimation method and apparatus, and electronic device, storage medium and computer program product

Abstract

The present disclosure relates to the technical field of computers, and relates to a network training method and apparatus for pose estimation, a pose estimation method and apparatus, and an electronic device, a storage medium and a computer program product. The network training method for pose estimation comprises: inputting a sample image, which is acquired from a training data set, into a preset feature extraction network; and performing training on an initial pose network according to a feature tensor, which is outputted by the feature extraction network, of a target object in the sample image, so as to obtain a target pose network. By means of the embodiments of the present disclosure, the training efficiency can be improved, thereby further reducing the consumption of computing power resources a processor and memory resources.

Description

Network training method and device for posture estimation, posture estimation method and device, electronic device, storage medium and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure is based on the Chinese patent application with application number 202311028342.X, application date August 15, 2023, and application name “Network training method and device for posture estimation, posture estimation method and device, electronic device, storage medium and computer program product”, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into the present disclosure as a reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a network training method and device for posture estimation, a posture estimation method and device, an electronic device, a storage medium, and a computer program product. Background Art

[0004] In many application areas, such as robot vision, motion tracking, and single-camera calibration, pose estimation of target objects in images is required, and subsequent business processing is based on the pose estimation results. For example, in face recognition scenarios, head pose estimation can be used to filter out qualified face images, and then recognition can be performed on the filtered face images. In augmented reality (AR) interaction scenarios, head pose estimation can be performed on captured face images to provide users with more realistic virtual effects (such as decorative effects such as hairpins). In safe driving detection scenarios, head pose estimation can be performed on the driver's face image to determine whether the driver is looking around.

[0005] Taking safe driving monitoring as an example, with the development of smart cockpit technology, the Driver Monitor System (DMS) has become an essential technology for smart cockpits due to its practicality and safety. DMS monitors the driver's fatigue, attention, and dangerous driving behavior in real time while driving, effectively reducing traffic accidents caused by subjective driver factors.

[0006] DMS fatigue detection not only uses facial features to determine eye openness and yawning, but also relies on the angle range of head posture to comprehensively determine the driver's concentration and whether they are dozing off. Furthermore, DMS distraction detection also uses head posture to preliminarily determine whether the driver is looking down or turning their head. Therefore, head posture estimation has become an indispensable component of DMS technology.

[0007] In related technologies, algorithms for estimating head posture consume a lot of computing power and memory resources.

[0008] Summary of the Invention

[0009] The present disclosure provides a network training method and device for posture estimation, a posture estimation method and device, an electronic device, a storage medium, and a computer program product.

[0010] According to one aspect of the present disclosure, a network training method for posture estimation is provided, comprising: obtaining a training data set; inputting sample images in the training data set into a preset feature extraction network to determine a feature tensor of a target object in the sample image; and training an initial posture network based on the feature tensor of the target object to obtain a target posture network, wherein the target posture network is used to determine the posture information of the target object in the image to be processed in combination with the feature extraction network.

[0011] In one possible implementation, the initial posture network is trained according to the feature tensor of the target object to obtain a target posture network, including: performing multiple rounds of iterative training on the initial posture network according to the feature tensor of the target object to obtain multiple alternative posture networks; when a training end condition is met, selecting a target posture network from the multiple alternative posture networks, the training end condition including that the number of iterative training rounds of the initial posture network is greater than or equal to a first preset threshold, and / or that the learning rate of the current round of iterative training is less than a second preset threshold.

[0012] In one possible implementation, the initial posture network is iteratively trained for multiple rounds based on the feature tensor of the target object to obtain multiple alternative posture networks, including: for any round of iterative training of the initial posture network, the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result; the posture loss is determined based on a preset posture loss function, the posture prediction result, and the annotation information of the training data set, wherein the posture loss is used to indicate the detection error of the initial posture network on the posture information of the target object; based on the posture loss, the initial posture network is iteratively trained to obtain the alternative posture network corresponding to the current round of iterative training.

[0013] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result, including: inputting the feature tensor of the target object into the first network to obtain a posture prediction result, and the posture prediction result includes the rotation matrix of the target object predicted by the first network; the preset posture loss function includes a first loss function, and the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network. According to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, the posture loss is determined, including: inputting the rotation matrix of the target object predicted by the first network and the annotation information of the training data set corresponding to the target object into the first loss function to determine the posture loss.

[0014] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object. The feature tensor of the target object is input into the initial posture network to obtain a posture prediction result, including: inputting the feature tensor of the target object into the first network and the second network respectively to obtain a posture prediction result, the posture prediction result including the rotation matrix of the target object predicted by the first network and the posture angle of the target object predicted by the second network; the preset posture loss function includes the product of a first loss function and a second loss function, the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network, and the second loss function is used to indicate the detection error of the posture angle of the target object predicted by the second network. According to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, the posture loss is determined, including: inputting the rotation matrix of the target object predicted by the first network, the posture angle of the target object predicted by the second network, and the annotation information of the training data set corresponding to the target object into the preset posture loss function to determine the posture loss.

[0015] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, the first network including a first network layer, a second network layer, and a third network layer; wherein the first network layer includes a convolution unit with a convolution kernel size of a first preset size, a normalization unit, and a nonlinear activation unit; the second network layer includes a depthwise separable convolution unit with a convolution kernel size of a second preset size, a normalization unit, and a nonlinear activation unit, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit.

[0016] In a possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object; the first network includes a first network layer, a second network layer, and a third network layer, and the second network includes a fourth network layer, a fifth network layer, a sixth network layer, and a seventh network layer; wherein the first network layer includes a convolution unit with a convolution kernel size of a first preset size, a normalization unit, and a nonlinear activation unit; the second network layer includes a depthwise separable convolution unit with a convolution kernel size of a second preset size, a normalization unit, and a nonlinear activation unit, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit; the fourth network layer includes a convolution unit with a convolution kernel size of the first preset size, a normalization unit, and a nonlinear activation unit; the fifth network layer includes a depthwise separable convolution unit with a convolution kernel size of the second preset size, a normalization unit, and a nonlinear activation unit; the sixth network layer includes a unit convolution unit, a normalization unit, and a nonlinear activation unit; and the seventh network layer includes a unit convolution unit.

[0017] In one possible implementation, obtaining a training data set includes: obtaining a first sample set, the first sample set including sample images in which a posture angle of a target object is greater than a preset angle threshold; obtaining a second sample set, the second sample set including sample images collected by multiple sensors in different scenarios; performing data set enhancement processing on the first sample set and / or the second sample set to obtain a third sample set; and determining a training data set based on the first sample set, the second sample set, and the third sample.

[0018] According to one aspect of the present disclosure, a posture estimation method is provided, comprising: acquiring an image to be processed; inputting the image to be processed into a preset feature extraction network to obtain a feature tensor of a target object in the image to be processed; inputting the feature tensor of the target object in the image to be processed into a target posture network for processing, and determining a detection result of the image to be processed, the detection result including posture information of the target object; wherein the target posture network is trained according to the above-mentioned network training method for posture estimation.

[0019] According to one aspect of the present disclosure, a network training device for posture estimation is provided, comprising: an acquisition module configured to acquire a training data set; a determination module configured to input a sample image in the training data set into a preset feature extraction network to determine a feature tensor of a target object in the sample image; and a training module configured to train an initial posture network according to the feature tensor of the target object to obtain a target posture network, wherein the target posture network is used to determine the posture information of the target object in the image to be processed in combination with the feature extraction network.

[0020] In one possible implementation, the training module is further configured to: perform multiple rounds of iterative training on the initial posture network according to the feature tensor of the target object to obtain multiple alternative posture networks; and select a target posture network from the multiple alternative posture networks when the training end conditions are met, and the training end conditions include that the number of iterative training rounds of the initial posture network is greater than or equal to a first preset threshold, and / or that the learning rate of the current round of iterative training is less than a second preset threshold.

[0021] In one possible implementation, the initial posture network is iteratively trained for multiple rounds based on the feature tensor of the target object to obtain multiple alternative posture networks, including: for any round of iterative training of the initial posture network, the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result; the posture loss is determined based on a preset posture loss function, the posture prediction result, and the annotation information of the training data set, wherein the posture loss is used to indicate the detection error of the initial posture network on the posture information of the target object; based on the posture loss, the initial posture network is iteratively trained to obtain the alternative posture network corresponding to the current round of iterative training.

[0022] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result, including: inputting the feature tensor of the target object into the first network to obtain a posture prediction result, and the posture prediction result includes the rotation matrix of the target object predicted by the first network; the preset posture loss function includes a first loss function, and the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network. According to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, the posture loss is determined, including: inputting the rotation matrix of the target object predicted by the first network and the annotation information of the training data set corresponding to the target object into the first loss function to determine the posture loss.

[0023] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object; the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result, including: inputting the feature tensor of the target object into the first network and the second network respectively to obtain a posture prediction result, the posture prediction result including the rotation matrix of the target object predicted by the first network and the posture angle of the target object predicted by the second network; the preset posture loss function includes the product of a first loss function and a second loss function, the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network, and the second loss function is used to indicate the detection error of the posture angle of the target object predicted by the second network, and the posture loss is determined according to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, including: inputting the rotation matrix of the target object predicted by the first network, the posture angle of the target object predicted by the second network, and the annotation information of the training data set corresponding to the target object into the preset posture loss function to determine the posture loss.

[0024] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, the first network including a first network layer, a second network layer, and a third network layer; wherein the first network layer includes a convolution unit with a convolution kernel size of a first preset size, a normalization unit, and a nonlinear activation unit; the second network layer includes a depthwise separable convolution unit with a convolution kernel size of a second preset size, a normalization unit, and a nonlinear activation unit, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit.

[0025] In a possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object; the first network includes a first network layer, a second network layer, and a third network layer, and the second network includes a fourth network layer, a fifth network layer, a sixth network layer, and a seventh network layer; wherein the first network layer includes a convolution unit with a convolution kernel size of a first preset size, a normalization unit, and a nonlinear activation unit; the second network layer includes a depthwise separable convolution unit with a convolution kernel size of a second preset size, a normalization unit, and a nonlinear activation unit, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit; the fourth network layer includes a convolution unit with a convolution kernel size of the first preset size, a normalization unit, and a nonlinear activation unit; the fifth network layer includes a depthwise separable convolution unit with a convolution kernel size of the second preset size, a normalization unit, and a nonlinear activation unit; the sixth network layer includes a unit convolution unit, a normalization unit, and a nonlinear activation unit; and the seventh network layer includes a unit convolution unit.

[0026] In one possible implementation, the acquisition module is configured to: acquire a first sample set, the first sample set including sample images in which the posture angle of the target object is greater than a preset angle threshold; acquire a second sample set, the second sample set including sample images collected by multiple sensors in different scenarios; perform data set enhancement processing on the first sample set and / or the second sample set to obtain a third sample set; and determine a training data set based on the first sample set, the second sample set, and the third sample.

[0027] According to one aspect of the present disclosure, a posture estimation device is provided, comprising: a module for acquiring an image to be processed, configured to acquire an image to be processed; a feature extraction module, configured to input the image to be processed into a preset feature extraction network to obtain a feature tensor of a target object in the image to be processed; a posture estimation module, configured to input the feature tensor of the target object in the image to be processed into a target posture network for processing, and determine a detection result of the image to be processed, wherein the detection result includes posture information of the target object; wherein the target posture network is trained according to the above-mentioned network training method for posture estimation.

[0028] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0029] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0030] According to one aspect of the present disclosure, a computer program product is provided, comprising computer-executable instructions, which implement the above method when executed by a processor.

[0031] In an embodiment of the present disclosure, the sample images in the acquired training data set can be input into a preset feature extraction network (e.g., a key point feature extraction network built into the driver monitoring system DMS) to train the initial posture network based on the feature tensor of the target object in the sample image output by the feature extraction network to obtain a target posture network. The target posture network trained in this way can be added to the already trained feature extraction network (e.g., a key point feature extraction network built into the driver monitoring system DMS) during network deployment in the inference phase without performing additional data preprocessing. The feature tensor output by the existing feature extraction network can be directly reused, which is beneficial to reducing the occupancy of computing resources and dependence on related algorithm libraries during deployment. Moreover, during the network training process, no feature extraction network-related training is required, which can improve training efficiency and further reduce the consumption of processor computing power resources and memory resources.

[0032] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0034] FIG1 is a schematic flow chart showing a network training method for posture estimation according to an embodiment of the present disclosure.

[0035] FIG2 is a schematic diagram showing attitude angles according to an embodiment of the present disclosure.

[0036] FIG3 shows a schematic diagram of an initial posture network according to an embodiment of the present disclosure.

[0037] FIG4 shows a schematic diagram of another initial posture network according to an embodiment of the present disclosure.

[0038] FIG5 is a schematic diagram showing a detection result determined by a posture estimation method according to an embodiment of the present disclosure.

[0039] FIG6 shows a block diagram of a network training device for pose estimation according to an embodiment of the present disclosure.

[0040] FIG7 shows a block diagram of a posture estimation apparatus according to an embodiment of the present disclosure.

[0041] FIG8 shows a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0043] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0044] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0045] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0046] In the related art, for pose estimation schemes of target objects (such as faces) based on key point technology, for example, the SolvePnP algorithm (a method that calculates the projection relationship between N feature points in the physical world and N image points in the image imaging to obtain the pose of the camera or object), the 3D Morphable Models (3DMM) algorithm, etc. These methods are relatively dependent on the accuracy of the key points of the target object. In addition, the pose estimation algorithm based on key points involves a large number of matrix calculations and iterative calculations, which requires a large amount of computing resources and consumes a lot of power, which may reduce the real-time performance of the pose estimation. For the pose estimation model scheme based on deep learning to establish the target object, in addition to deploying the key point model, an additional model for pose estimation must be deployed, which increases the use of computing resources and memory, affecting the inference speed of the pose estimation. For the pose estimation scheme based only on lightweight target objects, its accuracy is not high and its generalization ability is poor.

[0047] In view of this, the embodiments of the present disclosure can input the sample images in the acquired training data set into a preset feature extraction network, such as the key point feature extraction network built into the driver monitoring system DMS (also known as the backbone network of the key point extraction network), which can be used to extract features related to the key points of the target object, so as to train the initial posture network according to the feature tensor of the target object in the sample image output by the feature extraction network to obtain the target posture network. The target posture network trained in this way can be added to the already trained feature extraction network (such as the key point feature extraction network built into the driver monitoring system DMS) during the network deployment in the inference phase, without performing additional data preprocessing, and directly reuse the feature tensor output by the existing feature extraction network, which is conducive to reducing the occupancy of computing resources and the dependence on related algorithm libraries during deployment; moreover, during the network training process, no feature extraction network-related training is required, which can improve training efficiency and further reduce the consumption of processor computing power resources and memory resources.

[0048] FIG1 shows a flow chart of a network training method for posture estimation according to an embodiment of the present disclosure. As shown in FIG1 , the network training method for posture estimation includes: in step S11 , obtaining a training data set.

[0049] In step S12, the sample images in the training data set are input into a preset feature extraction network to determine the feature tensor of the target object in the sample image, and the network parameters of the feature extraction network remain fixed.

[0050] In step S13, the initial posture network is trained according to the feature tensor of the target object to obtain a target posture network, wherein the target posture network is used to determine the posture information of the target object in the image to be processed in combination with the feature extraction network.

[0051] In one possible implementation, the network training method for posture estimation can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the network training method for posture estimation can be executed by a server. In one example, the network training method for posture estimation can be applied to a cloud server or a local server. The cloud server can be a public cloud server or a private cloud server, which can be flexibly selected according to actual conditions.

[0052] In one possible implementation, the target object in the embodiments of the present disclosure can be any object whose posture needs to be determined, such as a human face, limbs, or other parts of the human body, or other object-related identifiers, landmarks, or other objects. For ease of explanation, the subsequent disclosed embodiments use a human head as the target object. The case where the target object is other objects can be flexibly expanded according to the subsequent disclosed embodiments, and this disclosure does not impose any restrictions on this.

[0053] In one possible implementation, a training dataset may be obtained in step S11. The training dataset may include a large number (e.g., millions) of sample images for training a posture network. Each sample image may correspond to a piece of annotation information indicating the posture information of a target object in the sample image, such as a rotation matrix and / or posture angle of the target object. The annotation information may include various forms such as text annotation, symbol annotation, digital annotation, and graphic annotation, and this disclosure does not impose any specific limitations on this.

[0054] In step S11, a training dataset is obtained. In step S12, a pre-trained feature extraction network is obtained, and the network parameters of the feature extraction network are frozen. Sample images from the training dataset are input into the feature extraction network, and features of the target object in the sample image are extracted to determine a feature tensor of the target object in the sample image. The feature tensor of the target object can be used to indicate the characteristics of the target object. For example, the feature tensor can be a multidimensional tensor that carries key point information of the target object. The present disclosure does not impose any specific restrictions on the size and form of the feature tensor.

[0055] Among them, freezing the network parameters of the feature extraction network means that the network parameters of the feature extraction network remain fixed. In this way, for the posture estimation network that includes the preset feature extraction network, since the network parameters of the feature extraction network remain fixed (the network parameters of the feature extraction network do not need to be trained again), the training data set is input into the posture estimation network to train the posture estimation network. In essence, the feature tensor output by the feature extraction network with fixed network parameters is used to train the initial posture network. This can more efficiently reuse the existing trained feature extraction network, effectively reduce the number of parameters to be adjusted in the posture estimation network, help improve training efficiency, and further reduce the consumption of processor computing power resources and memory resources.

[0056] For example, in a driver monitoring system (DMS) scenario, the target object is a human head. Since head pose can be estimated and calculated based on facial key points, the DMS's key point feature extraction network can be reused. Sample images from the training dataset are input into the DMS's key point feature extraction network to extract features from the face in the sample images and determine a feature tensor for the face in the sample images. The feature extraction network used to extract features from facial key points can be flexibly selected based on actual circumstances. Any trained neural network capable of extracting features from facial images can be used as an implementation of step S12, and this disclosure does not impose any restrictions on this.

[0057] Exemplarily, the preset feature extraction network can be any neural network with a feature extraction function for key points of the target object, and its implementation form is not limited in the embodiments of the present disclosure. The preset feature extraction network can be any common type of trained neural network model. In one possible implementation, the preset feature extraction network can include at least one of the following: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Back Propagation Neural Networks (BP), Backbone Neural Networks (BNN), etc.

[0058] Exemplarily, the preset feature extraction network can be a neural network that has the function of extracting features of key points of target objects in an image, constructed in any form through different convolutional layers, pooling layers, classification layers or other types of network layers.

[0059] It should be understood that the network structure of the above-mentioned feature extraction network is only an example, and its specific network structure can be determined according to actual conditions. The above-mentioned example does not constitute a limitation on the embodiments of the present disclosure.

[0060] In step S12, a preset feature extraction network can be used to determine the feature tensor of the target object in the sample image. In step S13, the feature tensor corresponding to each sample image in the data set can be individually input into the initial pose network to train the initial pose network to obtain a trained target pose network. Alternatively, multiple feature tensors corresponding to multiple sample images can be input into the initial pose network in batches as batch data to train the initial pose feature detection network to obtain a trained target pose network. The present disclosure does not impose any restrictions on this. The error loss of the initial pose network can be determined based on the prediction results of the initial pose network and the annotation information of the sample images in the training set, and the initial pose network can be trained based on the error loss of the initial pose network to obtain a target pose network.

[0061] Exemplarily, before training the initial posture network, a posture network can be constructed first. The posture network can be a neural network with posture estimation function constructed in any form through different convolutional layers, pooling layers, classification layers or other types of network layers. In one possible implementation, the posture network can be a lightweight network whose network scale is smaller than a preset scale; the network scale of the posture network can include at least one of the number of network parameters of the posture network, the number of layers of the posture network and the number of channels of the network layer of the posture network. For example, the number of network parameters, the number of network layers and the number of channels of the posture network can be smaller than a preset scale threshold. The specific situation can be flexibly determined according to the actual situation, and the present disclosure does not impose specific restrictions on this.

[0062] After constructing the pose network, the network parameters of the pose network can be initialized (for example, by setting the network parameters to random values ​​within a preset threshold range) to obtain an initial pose network. The initial pose network can then be trained based on the feature tensor of the target object, and the network parameters of the initial pose network can be continuously adjusted until training is completed, resulting in a target pose network.

[0063] For example, the trained target pose network can be combined with a pre-set feature extraction network to determine the pose information of the target object in the image to be processed. For example, in a driver monitoring system (DMS) scenario, the DMS can use its own keypoint feature extraction network to extract the feature tensor of the driver's face. This feature tensor is then fed into the trained target pose network for processing to determine the pose information of the driver's head.

[0064] Exemplarily, the posture information may include posture angles. Figure 2 shows a schematic diagram of posture angles according to an embodiment of the present disclosure. As shown in Figure 2, assuming that the target object is a human head, the posture angles of the head may include three angles, namely a pitch angle 21 (pitch) rotating around the x-axis of space, a yaw angle 22 (yaw) rotating around the y-axis of space, and a roll angle 23 (roll) rotating around the z-axis of space. In spatial solid geometry, the x-axis, y-axis, and z-axis are perpendicular to each other.

[0065] For example, the posture information may also include a rotation matrix (e.g., a 3×3 matrix), and the posture angle and the rotation matrix may be converted to each other. For example, assuming that the posture angle θ of the target object is [θ x ,θ y ,θ z ], where θ x represents the pitch angle of the target object, θ y represents the yaw angle of the target object, θ z Represents the roll angle of the target object. The rotation matrix R(θ x ,θ y ,θ z ) can be obtained by the posture angle θ=[θ x ,θ y ,θ z ] means:

[0066] Similarly, when the rotation matrix of the target object is known, the attitude angle of the target object can be obtained by inversely solving formula (1).

[0067] Through steps S11 to S13, the sample images in the acquired training data set can be input into a preset feature extraction network (e.g., a key point feature extraction network built into the driver monitoring system (DMS)). Based on the feature tensor of the target object in the sample image output by the feature extraction network, the initial pose network is trained to obtain a target pose network. The target pose network trained in this manner can be added to an already trained feature extraction network (e.g., a key point feature extraction network built into the driver monitoring system (DMS)) during network deployment in the inference phase. No additional data preprocessing is performed, and the feature tensors output by the existing feature extraction network can be directly reused. This helps reduce the use of computing resources and reliance on related algorithm libraries during deployment. Furthermore, during network training, no further feature extraction network-related training is required, which improves training efficiency and further reduces the consumption of processor computing power and memory resources.

[0068] The following describes a network training method for posture estimation according to an embodiment of the present disclosure.

[0069] In one possible implementation, step S11 of obtaining a training data set may include: obtaining a first sample set, the first sample set including sample images in which the posture angle of the target object is greater than a preset angle threshold; obtaining a second sample set, the second sample set including sample images collected by multiple sensors in different scenarios; performing data set enhancement processing on the first sample set and / or the second sample set to obtain a third sample set; and determining a training data set based on the first sample set, the second sample set, and the third sample.

[0070] Exemplarily, one or more publicly available data sets (including the rich large-angle head posture data set 300W-LP) can be used as the first sample set. The first sample set may include sample images of various angles, including not only images with posture angles less than or equal to a preset angle threshold (for example, 15°), but also images with posture angles greater than the preset angle threshold (for example, 15°).

[0071] Among them, if the sample image lacks annotation information related to the posture information, such as the rotation matrix and / or posture angle of the target object, the target object of the sample image can be annotated accordingly; or, if the annotation information is the posture angle, the corresponding rotation matrix can be generated according to the posture angle, and the rotation matrix can also be saved as the annotation information; or, if the annotation information is the rotation matrix, the corresponding posture angle can be generated according to the rotation matrix, and the posture angle can also be saved as the annotation information.

[0072] Exemplarily, in order to increase the diversity of sample images in the training dataset, thereby improving the generalization and accuracy of the subsequent training model, the second sample set may include sample images collected by multiple sensors in different scenarios. The sensors used as sample image acquisition devices may include depth cameras, camera arrays, monocular cameras, cameras, infrared cameras, CCD (charge coupled) image sensors, CMOS (complementary metal oxide conductor) image sensors, etc. The shooting scenes may include any natural scene in the real world, artificially arranged experimental scenes, scenes under different light conditions at different time periods, etc., wherein a natural scene is any scene in the real world (e.g., work, study, life), for example, including images captured in parks, offices, schools, shopping malls, etc. The backgrounds of images captured in natural scenes can be diverse. The present disclosure does not limit the acquisition devices, shooting angles, and shooting scenes of the sample images. In the example, the second sample set may also include one or more existing datasets containing diverse data (e.g., face recognition data MS1M), which have been disclosed, and the present disclosure does not impose specific restrictions on this.

[0073] Among them, since the sample images collected by multiple sensors in different scenes do not have annotation information of the posture of the target object (such as the head), the large model posture angle annotation model (such as the 6DRepNet model) trained by the first sample set can be used to annotate the sample images with posture angles (pitch angle, yaw angle, roll angle). During the annotation process, the sample images to be annotated can be randomly offset and randomly scaled N times (for example, N = 12, the present disclosure is for N) to generate N groups of posture angles. The maximum and minimum values ​​in the N groups of posture angles can be removed, and the remaining posture angles are averaged to obtain the posture angles of the sample images. Then, P (for example, P = 1 million) sample images with posture angles greater than a preset angle threshold (for example, 15°) and P sample images with postures less than a preset angle threshold can be extracted, and then the corresponding rotation matrix is ​​generated according to the posture angle, and the rotation matrix is ​​also saved as annotation information to obtain the second sample set. By setting the preset angle threshold, the data distribution range selected by the second sample set can be balanced.

[0074] Exemplarily, the first sample set and / or the second sample set are also subjected to dataset enhancement processing, such as random image translation, random scaling, random color change, random channel conversion, grayscale change, horizontal flipping, etc., to further increase the diversity of the training dataset and improve the generalization of the model.

[0075] For example, the image path of the sample image in the first sample set and / or the second sample set and the annotation information corresponding to the sample image can be loaded, and the annotation information may include the posture angle θ of the annotated target object. gt and / or the rotation matrix R gt Online dataset enhancement processing can be performed on the sample images in the first sample set and / or the second sample set, such as random color change with a probability of 0.5, random channel conversion or grayscale change with a probability of 0.3, horizontal flip with a probability of 0.5, random image translation or random scaling with a probability of 0.4, etc., to obtain a third sample set. It should be understood that the present disclosure does not limit the specific method of dataset enhancement processing.

[0076] In this way, a large-scale training dataset containing rich large-angle target objects can be constructed, which is conducive to the subsequent use of multiple large-scale data training, so that the trained target posture network has better accuracy and generalization.

[0077] After the training dataset is constructed in step S11, the feature extraction network with frozen network parameters preset in step S12 and the initial pose network whose network parameters need to be optimized can be connected in series, that is, the output of the feature extraction network is connected to the input of the initial pose network to form a pose estimation network. Since the network parameters of the feature extraction network part in the pose estimation network are frozen, when the training dataset is input into the pose estimation network for training, in essence, the initial pose network is trained using the feature tensor output by the feature extraction network with frozen network parameters.

[0078] Therefore, in step S12, the sample images in the training dataset are input into the preset feature extraction network to determine the feature tensor of the target object in the sample images. In step S13, the initial pose network can be trained based on the feature tensor of the target object to obtain the target pose network.

[0079] In a possible implementation manner, to improve the training accuracy and obtain a target pose network with higher accuracy, step S13 may include: performing multiple rounds of iterative training on the initial pose network according to the feature tensor of the target object to obtain multiple candidate pose networks. When the training end condition is satisfied, the target pose network is selected from the multiple candidate pose networks. Among them, each round of iterative training corresponds to a candidate pose network, and the network parameters of the candidate pose networks trained in each round may be the same or different. The training end condition may include that the number of iterative training rounds of the initial pose network is greater than or equal to the first preset threshold, and / or the learning rate of the current round of iterative training is less than the second preset threshold. The present disclosure does not limit the specific values of the first preset threshold and the second preset threshold.

[0080] Exemplarily, assume that the training dataset includes X sample images. During each round of iterative training, all X sample images in the training dataset can be respectively input into the feature extraction network, and the initial pose network is trained using the feature tensors corresponding to all X sample images in the training dataset to obtain the candidate pose network corresponding to each round. Alternatively, during each round of iterative training, Y sample images (Y < X) can also be randomly selected from the X sample images in the training dataset, and the initial pose network is trained using the feature tensors corresponding to some sample images (such as Y sample images) in the training dataset to obtain the candidate pose network corresponding to each round.

[0081] Among them, during the process of performing multiple rounds of iterative training on the initial pose network, the multiple rounds of iterative training on the initial pose network can be performed in parallel or serially (round by round), and the present disclosure does not limit this.

[0082] In practical applications, the network parameters of the initial posture network can be used as optimization parameters, and the learning rate of the initial posture network can be set to 10 -3 When the pose loss output by the initial pose network (which can be determined by a preset pose loss function, the pose prediction results predicted by the initial pose network, and the annotation information of the training dataset) stops decreasing for 30 consecutive rounds, the learning rate is decayed, and the learning rate decay rate is 0.1. It should be understood that the various values ​​set above are for example only and can be adjusted according to actual application scenarios. This disclosure does not limit them.

[0083] When the training end conditions are met, for example, the number of iterative training rounds of the initial posture network is greater than or equal to a first preset threshold (for example, 200), and / or the learning rate of the current round of iterative training is less than a second preset threshold (for example, 10 -6 ), the training can be terminated, and a target pose network can be selected from the multiple candidate pose networks. The candidate pose network with the smallest pose loss can be selected as the target pose network from the multiple candidate pose networks. Alternatively, the same test image can be input into multiple candidate pose networks, and the candidate pose network with the fastest processing speed and / or the most accurate pose estimation can be selected as the target pose network. The present disclosure does not impose any specific restrictions on the method of selecting the target pose network from the multiple candidate pose networks.

[0084] In this way, the utilization rate of the training data set can be improved. The feature tensor corresponding to the same sample object in the training data set can be passed multiple times in the initial posture network (possibly once in each round) to obtain multiple alternative posture networks, and the optimal alternative posture network can be used as the target posture network, further improving the accuracy of the target posture network.

[0085] In one possible implementation, for any round of iterative training of the initial posture network, the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result; the posture loss is determined based on a preset posture loss function, the posture prediction result, and the annotation information of the training data set, wherein the posture loss is used to indicate the detection error of the initial posture network on the posture information of the target object; based on the posture loss, the initial posture network is iteratively trained to obtain an alternative posture network corresponding to the current round of iterative training.

[0086] For example, for any round (e.g., the current round) of iterative training of the initial posture network, M (M≤X) sample images can be randomly selected from the training data set (a total of X sample images), and the annotation information corresponding to each sample image can include: posture angle θ gt and / or the rotation matrix R gt .

[0087] M sample images can be fed into the feature extraction network with frozen network parameters to obtain the feature tensors corresponding to the M sample images, and the feature tensors corresponding to the M sample images can be fed into the initial posture network to obtain the posture prediction results, which may include the predicted rotation matrix R p and / or attitude angle θ p .

[0088] The rotation matrix R included in the posture prediction result can be p , the annotation information of the sample image includes the rotation matrix R gt , input the preset posture loss function, and determine the function value of the posture loss function as the posture loss. The posture loss is used to measure the rotation matrix R included in the posture prediction result. p The annotation information of the sample image includes the rotation matrix R gt The deviation of the posture prediction result includes the rotation matrix R p The closer the annotation information of the sample image is, the more rotation matrix R gt , the smaller the posture loss.

[0089] Similarly, the posture angle θ included in the posture prediction result can also be p , the annotation information of the sample image includes the posture angle θ gt , input the preset posture loss function, and determine the function value of the posture loss function as the posture loss. The posture loss is used to measure the posture angle θ included in the posture prediction result. p The pose angle θ included in the annotation information of the sample image gt The deviation of the posture prediction result includes the posture angle θ p The closer the annotation information of the sample image is, the more posture angle θ gt , the smaller the posture loss.

[0090] It should be understood that, considering that the attitude angle and the rotation matrix can be converted to each other, the rotation matrix R included in the attitude prediction result can also be converted to p , the annotation information of the sample image includes the posture angle θ p ; Or, the posture angle θ included in the posture prediction result p , the annotation information of the sample image includes the rotation matrix R gt , input a preset posture loss function to determine it as the posture loss. The present disclosure does not limit the specific method of determining the posture loss.

[0091] According to the posture loss, the initial posture network is iteratively trained until the initial posture network converges, and the candidate posture network corresponding to the round (e.g., the current round) of iterative training is obtained.

[0092] Among them, the network parameters of the initial posture network can be adjusted by the gradient descent method to optimize the network parameters and improve the classification accuracy; among them, the gradient descent method can adopt the stochastic gradient descent method (Stochastic Gradient Descent, SGD), the batch gradient descent method (Batch Gradient Descent, BGD) or the mini batch gradient descent method (Mini Batch Gradient Descent, MBGD), and the present disclosure does not limit the selection type of the gradient descent method.

[0093] Among them, when the posture prediction result output by the initial posture network is quite different from the annotation information of the sample image, back propagation can be used to calculate the partial derivative of the posture loss function with respect to the network parameters of each layer in the initial posture network layer by layer, and construct the gradient of the posture loss function with respect to the network parameters of each layer, until the deviation between the posture prediction result output by the initial posture network and the annotation information of the sample image reaches the predetermined target range.

[0094] In this way, the alternative posture network corresponding to each round of iterative training can be obtained more accurately and efficiently.

[0095] The following describes the training process of the posture network according to the embodiment of the present disclosure by taking posture networks of two different structures as examples.

[0096] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object. Inputting the feature tensor of the target object into the initial posture network to obtain a posture prediction result includes: inputting the feature tensor of the target object into the first network to obtain a posture prediction result, wherein the posture prediction result includes the rotation matrix of the target object predicted by the first network. The preset posture loss function includes a first loss function, which is used to indicate the detection error of the rotation matrix of the target object predicted by the first network. The posture loss is determined based on the preset posture loss function, the posture prediction result, and the annotation information of the training data set, including: inputting the rotation matrix of the target object predicted by the first network and the annotation information of the training data set corresponding to the target object into the first loss function to determine the posture loss.

[0097] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, the first network including a first network layer, a second network layer, and a third network layer; wherein the first network layer includes a convolution unit with a convolution kernel size of a first preset size, a normalization unit, and a nonlinear activation unit; the second network layer includes a depthwise separable convolution unit with a convolution kernel size of a second preset size, a normalization unit, and a nonlinear activation unit, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit.

[0098] In this example, the feature tensor of the target object can be input into the first network layer, and the feature tensors output by the feature extraction network can be fused and calculated to obtain the processing results of the first network layer. The processing results output by the first network layer can be input into the second network layer, and the spatial position information of each channel of the processing results output by the first network layer can be fused through the second network layer to obtain the processing results of the second network layer. The processing results output by the second network layer can be input into the third network layer, and the number of channels can be adjusted through the third network layer to obtain the predicted rotation matrix of the target object.

[0099] The convolution unit is configured to provide a convolution operation for the input data, with a convolution kernel size of a first preset size. The value of the first preset size may be set based on the actual application scenario, and this disclosure does not impose any restrictions on this. If the feature tensor of the target object is multi-channel, the convolution unit's convolution kernel may be convolved with each channel of the feature tensor separately, and the convolution results of each channel may be added together to obtain a convolution result.

[0100] The depthwise separable convolution unit is used to provide a depthwise separable convolution operation with a convolution kernel size of a second preset size for the input data. When the second preset size is larger than the first preset size, the value of the second preset size can be set according to the actual application scenario, and the present disclosure does not impose any restrictions on this. The depthwise separable convolution can be divided into two processes: channel-by-channel convolution and point-by-point convolution. The information of different channels can be fused by calculating the weighted sum of elements at the same position in different channels. The depthwise separable convolution effectively reduces the number of parameters and computational cost by decomposing the convolution operation.

[0101] The normalization unit is used to constrain the distribution of data, thereby enhancing the generalization ability of the network. The normalization unit can be a batch normalization unit (Batch Norm, BN), layer normalization (LayerNorm, LN), instance normalization (Instance Norm, IN), group normalization (Group Norm, GN), etc., and this disclosure does not limit this.

[0102] The nonlinear activation unit is used to add some nonlinear features to the initial posture network, suppress noise, and make the initial posture network easier to fit the data. The nonlinear activation unit can be a linear rectified function (Rectified Linear Unit, ReLu), an sigmoid activation function (Sigmoid), a hyperbolic tangent function (Tanh), etc., and this disclosure does not limit this.

[0103] The network structure of the initial posture network in the embodiment of the present disclosure is simple, easy to implement, and can significantly reduce the consumption of hardware resources.

[0104] For example, FIG3 shows a schematic diagram of an initial posture network according to an embodiment of the present disclosure. As shown in FIG3 , the rotation matrix R used to predict the target object can be p The first network 30 is used as the initial posture network.

[0105] The first network 30 may be composed of three network layers. The first network layer 31 may include a convolution unit (Convolution) with a convolution kernel of 3×3, a batch normalization unit (BN), and a nonlinear activation unit (e.g., ReLU). The first network layer 31 may be used to perform a fusion calculation on the feature tensor f (e.g., a feature tensor having 7 elements in the length dimension and 7 elements in the width dimension) output by the feature extraction network 35 to extract information useful for the target object's posture. It should be understood that the convolution kernel size of 3×3 is only an example of the first preset size, and the present disclosure does not impose any specific limitation on the value of the first preset size.

[0106] The second network layer 32 may include a depthwise separable convolution unit (GDConv) with a 7×7 convolution kernel, a batch normalization unit (BN), and a nonlinear activation unit (e.g., ReLU). These units can be used to fuse positional information from each feature layer, and have fewer parameters and faster inference speed than convolution units. It should be understood that the 7×7 convolution kernel size is merely an example of the second preset size, and this disclosure does not impose any specific restrictions on the value of the second preset size.

[0107] The third network layer 33 may include a convolution unit with a convolution kernel of 1×1, that is, a unit convolution unit, which is used to adjust the number of channels and output 6 values ​​for posture estimation. Due to the orthogonal characteristics of the rotation matrix, the singular value decomposition can be performed by the Gram-Schmidt orthogonalization method, and the rotation matrix R can be represented by 6 parameters. n .

[0108] Statistics show that the first network 30 has a network parameter count of 0.302M, which is less than the 0.476M network parameter count of the lightweight network model MobileNetv1-0.5 in the related art. The multiplication and addition computational complexity is 28.94M. Therefore, the first network 30 consumes very little hardware resources. In subsequent actual deployments, the intermediate layer variables of the first network 30 can be two feature tensor data, which also consumes very little hardware resource memory.

[0109] As shown in FIG3 , the feature tensor f of the target object in the sample image 34 output by the feature extraction network 35 can be obtained, and the feature tensor f of the target object can be input into the first network 30 to obtain a posture prediction result, which can include the rotation matrix R of the target object predicted by the first network 30. p .

[0110] The first loss function Loss geodesic As a pose loss function Loss pose , which can be expressed as:

[0111] In formula (2), R p is the rotation matrix of the target object predicted by the first network 30, R gt is the rotation matrix contained in the annotation information of the sample image 34 corresponding to the target object, M represents the batch number of data (for example, M feature tensors f can be input to the first network 30 in each batch), m represents the mth data in the batch data, and tr represents the trace operation of the matrix. Since the posture angle and the rotation matrix can be converted to each other, if the annotation information in the sample image 34 only contains the posture angle θ gt , according to the formula (1) introduced above, the attitude angle θ gt Generate the rotation matrix R gt .

[0112] Due to the orthogonal characteristics of the rotation matrix, the singular value decomposition can be performed by the Gram-Schmidt orthogonalization method, that is, the rotation matrix R is represented by 6 parameters. p In this case, the first network 30 can represent the predicted rotation matrix R by predicting 6 parameters. p , which can be expressed as: R p =F GS ([a1 a2])=[b1 b2 b3] (3);

[0113] Among them, a1, a2, b1, b2, b3∈R 1×3That is, a1, a2, b1, b2, and b3 are all vectors with three rows and one column, [a1 a2] is a vector with three rows and two columns, and has 6 parameters, which are the 6 parameters predicted by the first network 30, and [b1 b2 b3] is a rotation matrix R with three rows and three columns. p , so through the transformation relationship F GS , the 6 parameters [a1 a2] predicted by the first network 30 can be converted into a 3×3 rotation matrix R p The conversion relationship between a1, a2 and b1, b2, b3 vectors F GS See formula (4), that is:

[0114] Among them, || || represents the norm operation, the dot product · is the inner product operation, and the cross product × is the outer product operation. In this way, the [a1 a2] containing 6 parameters output by the first network 30 is converted through a small number of transformation relations F GS Calculation can generate a rotation matrix R containing 9 parameters p .

[0115] In this way, since the change of the target object's posture angle is directly related to the rotation matrix of the target object's posture, compared with directly regressing the target object's posture angle containing 3 parameters, the first loss function Loss is geodesic As the first loss function, by predicting the rotation matrix containing 9 parameters, the accuracy is higher and the network is easier to converge.

[0116] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object. The feature tensor of the target object is input into the initial posture network to obtain a posture prediction result, including: inputting the feature tensor of the target object into the first network and the second network respectively to obtain a posture prediction result, and the posture prediction result includes the rotation matrix of the target object predicted by the first network, and the posture angle of the target object predicted by the second network.

[0117] The preset posture loss function includes the product of a first loss function and a second loss function, the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network, and the second loss function is used to indicate the detection error of the posture angle of the target object predicted by the second network. The rotation matrix of the target object predicted by the first network, the posture angle of the target object predicted by the second network, and the annotation information of the training data set corresponding to the target object are input into the preset posture loss function to determine the posture loss.

[0118] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object; the first network includes a first network layer, a second network layer, and a third network layer, and the second network includes a fourth network layer, a fifth network layer, a sixth network layer, and a seventh network layer.

[0119] Among them, the first network layer includes a convolution unit, a normalization unit, and a nonlinear activation unit with a convolution kernel size of a first preset size; the second network layer includes a depth-separable convolution unit, a normalization unit, and a nonlinear activation unit with a convolution kernel size of a second preset size, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit; the fourth network layer includes a convolution unit, a normalization unit, and a nonlinear activation unit with a convolution kernel size of a first preset size; the fifth network layer includes a depth-separable convolution unit, a normalization unit, and a nonlinear activation unit with a convolution kernel size of a second preset size, and the second preset size is larger than the first preset size; the sixth network layer includes a unit convolution unit, a normalization unit, and a nonlinear activation unit; the seventh network layer includes a unit convolution unit.

[0120] It can be seen that the network structure of the first network layer in the first network and the fourth network layer in the second network is the same, the network structure of the second network layer in the first network and the fifth network layer in the second network is the same, and the network structure of the third network layer in the first network and the seventh network layer in the second network is the same.

[0121] Therefore, compared to the first network, the second network of the embodiment of the present disclosure adds a sixth network layer. In this way, the second network can be constructed more efficiently and conveniently. The structure of the second network is simple and easy to implement, which greatly reduces the consumption of hardware resources. Among them, the convolution unit, depthwise separable convolution unit, normalization unit, and nonlinear activation unit of the initial posture network composed of the first network and the second network can be referred to above.

[0122] In this example, the feature tensor of the target object can be input into the first network layer, and the feature tensors output by the feature extraction network can be fused and calculated to obtain the processing results of the first network layer. The processing results output by the first network layer can be input into the second network layer, and the spatial position information of each channel of the processing results output by the first network layer can be fused through the second network layer to obtain the processing results of the second network layer. The processing results output by the second network layer can be input into the third network layer, and the number of channels can be adjusted through the third network layer to obtain the predicted rotation matrix of the target object.

[0123] Synchronously, the feature tensor of the target object can be input into the fourth network layer, and the feature tensor output by the feature extraction network can be fused and calculated to obtain the processing result of the fourth network layer. The processing result output by the fourth network layer can be input into the fifth network layer, and the spatial position information of each channel of the processing result output by the fourth network layer can be fused through the fifth network layer to obtain the processing result of the fifth network layer. The processing result output by the fifth network layer can be input into the sixth network layer, and the nonlinear fitting capability of the second network can be further increased through the sixth network layer to obtain the processing result of the sixth network layer. The processing result output by the sixth network layer can be input into the seventh network layer, and the number of channels can be adjusted through the seventh network layer to obtain the predicted posture angle of the target object.

[0124] The network structure of the initial posture network in the embodiment of the present disclosure is simple, easy to implement, and can significantly reduce the consumption of hardware resources.

[0125] For example, FIG4 shows a schematic diagram of another initial posture network according to an embodiment of the present disclosure. As shown in FIG4 , the initial posture network may include a rotation matrix R for predicting the target object. p The first network 30, and the pose angle θ for predicting the target object p The second network 40. The introduction of the first network 30 is shown in the relevant description of FIG3 above.

[0126] Exemplarily, the second network 40 may be composed of four network layers: a fourth network layer 41, a fifth network layer 42, a sixth network layer 43, and a seventh network layer 44. The fourth network layer 41 may include a convolution unit (Convolution) with a convolution kernel of 3×3, a batch normalization unit (BN), and a nonlinear activation unit (e.g., ReLU), which can be used to perform a fusion calculation on the feature tensor f output by the feature extraction network (e.g., a feature tensor with 7 elements in the length dimension and the width dimension, respectively), and extract information useful for the posture of the target object. It should be understood that the convolution kernel size of 3×3 is only an example of the first preset size, and the present disclosure does not specifically limit the value of the first preset size.

[0127] The fifth network layer 42 may include a depthwise separable convolution unit (GDConv) with a 7×7 convolution kernel, a batch normalization unit (BN), and a nonlinear activation unit (e.g., ReLU). These units can be used to fuse positional information from each feature layer, and have fewer parameters and faster inference speed than convolution units. It should be understood that the 7×7 convolution kernel size is merely an example of the second preset size, and this disclosure does not impose any specific restrictions on the value of the second preset size.

[0128] The sixth network layer 43 may include a convolution unit (Convolution) with a convolution kernel of 1×1, a batch normalization unit (BN), and a nonlinear activation unit (such as ReLU), which can be used to increase the nonlinear fitting capability of the second network 40.

[0129] The seventh network layer 44 may include a convolution unit with a convolution kernel of 1×1, which is used to adjust the number of channels and output the posture angle θ p .

[0130] As shown in FIG4 , the feature tensor f of the target object in the sample image 34 output by the feature extraction network 35 can be obtained, and the feature tensor f of the target object is input into the first network 30 and the second network 40 to obtain the posture prediction result, which includes the rotation matrix R of the target object predicted by the first network 30. p , and the pose angle θ of the target object predicted by the second network 40 p .

[0131] For the initial posture network shown in Figure 4, the posture loss function Loss pose It can be expressed as:

[0132] In formula (5), M represents the batch number of data (for example, M feature tensors f can be input to the first network 30 and the second network 40 respectively in each batch), m represents the mth data in the batch data, and Loss geodesic Represents the first loss function, which can be found in formula (2). aux represents the second loss function, which can be expressed as:

[0133] In formula (6), K=3 is the number of attitude angles, The k-th pose angle of the target object predicted by the second network 40, for example, the first (k=1) is the predicted pitch angle, the second one (k=2) is the predicted yaw angle, the third one (k=3) is the predicted roll angle; The k-th posture angle contained in the annotation information corresponding to the target object, for example, the first (k=1) is the pitch angle marked in the sample image 34, the second one (k=2) is the yaw angle marked in sample image 34, the third one (k=3) is the roll angle marked in the sample image 34.

[0134] Considering that in the training data set, the posture angle of the target object follows a normal distribution, that is, the larger the posture angle, the less data. In view of this, we can introduce a second network with an auxiliary role and its corresponding second loss function Loss aux In this way, during training, a smaller learning rate can be given to the second network, so that the second network can more easily converge on sample images with small angles, and the prediction deviation of sample images with large angles is larger. The corresponding second loss function Loss aux The greater the loss value is, the greater the loss value is. Therefore, the second loss function Loss aux With the first loss function Loss geodesic Multiply, the resulting posture loss function Loss pose , used as the training of the initial posture network, can adaptively increase the weight ratio of difficult samples in the optimization process (such as sample images with large angles of the target object) in the loss, improve the posture estimation accuracy of the trained target posture network for large-angle images, and further improve the accuracy of the target posture network in predicting the target object posture.

[0135] After obtaining the trained target posture network, the target posture network can be used in conjunction with a preset feature extraction network to determine the posture information of the target object in the image to be processed.

[0136] In a possible implementation, an embodiment of the present disclosure also provides a posture estimation method, which may include: obtaining an image to be processed; inputting the image to be processed into a preset feature extraction network to obtain a feature tensor of the target object in the image to be processed; inputting the feature tensor of the target object in the image to be processed into a target posture network for processing, and determining a detection result of the image to be processed, wherein the detection result includes the posture information of the target object; wherein the target posture network is trained according to the network training method for posture estimation introduced in steps S11 to S13 above.

[0137] Among them, the method of obtaining the image to be processed is not limited in the embodiment of the present disclosure. For example, the image to be processed can be obtained by reading a database storing the image to be processed, or the target object can be collected in certain scenarios to obtain the image to be processed, or a segment can be cut out or sampled from a video containing the target object to obtain the image to be processed, etc. The specific method of obtaining can be flexibly determined according to the actual situation.

[0138] The acquired image to be processed can be input into a preset feature extraction network (for example, the key point feature extraction network built into the DMS system, which can be used to extract features related to the key points of the target object) to obtain a feature tensor of the target object in the image to be processed, and the feature tensor is input into the target posture network trained using the method introduced in steps S11 to S13 above for processing to determine the detection result of the image to be processed, which includes the posture information of the target object.

[0139] FIG5 is a schematic diagram illustrating a detection result determined by a pose estimation method according to an embodiment of the present disclosure. As shown in FIG5 , the target object is a human face, the image to be processed is a facial image, and the coordinate axes in the facial image are used to visualize the pose angles of the face. Specifically, the first and second facial images 51 and 52 are RGB images, the third and fourth facial images 53 and 54 are near-infrared facial images, and the fifth facial image 55 is a cartoon image. This demonstrates that the pose estimation method according to an embodiment of the present disclosure not only has high accuracy but also good generalizability across different scenarios.

[0140] In summary, the embodiments of the present disclosure can input the sample images in the acquired training data set into a preset feature extraction network (such as the key point feature extraction network built into the driver monitoring system DMS, which can be used to extract features related to the key points of the target object) to train the initial posture network according to the feature tensor of the target object in the sample image output by the feature extraction network to obtain the target posture network. The target posture network trained in this way can be added to the already trained feature extraction network (such as the key point feature extraction network built into the driver monitoring system DMS) during the network deployment in the inference stage without performing additional data preprocessing. The feature tensor output by the existing feature extraction network can be directly reused, which is conducive to reducing the occupancy of computing resources and dependence on related algorithm libraries during deployment; moreover, during the network training process, there is no need to perform training related to the feature extraction network, which can improve training efficiency and further reduce the consumption of processor computing power resources and memory resources.

[0141] Furthermore, the initial posture network of the embodiment of the present disclosure can adopt a lightweight network model with few intermediate layer variables, which greatly reduces the consumption of hardware computing resources and memory resources. Moreover, the embodiment of the present disclosure also integrates the first loss function Loss ggeodesic And the second loss function Loss aux , propose a new focus optimization posture loss Loss pose, see formula (5). This makes the optimization process focus more on samples with small data volumes and large angles, increases the optimization weight of difficult samples, and improves the accuracy of the trained target pose network in predicting the pose of the target object. In addition, the training process of the embodiment of the present disclosure uses a large-scale training dataset (including a rich variety of large angles), making the optimized target pose network more accurate and generalizable.

[0142] It is understood that the various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Those skilled in the art will understand that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0143] In addition, the present disclosure also provides a network training device for posture estimation, a posture estimation method and device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any network training method for posture estimation provided by the present disclosure, and the corresponding technical solutions and descriptions are referred to the corresponding records in the method section.

[0144] Figure 6 shows a block diagram of a network training device for posture estimation according to an embodiment of the present disclosure. As shown in Figure 6, the device includes: an acquisition module 61, configured to acquire a training data set; a determination module 62, configured to input a sample image in the training data set into a preset feature extraction network to determine the feature tensor of the target object in the sample image; a training module 63, configured to train the initial posture network according to the feature tensor of the target object to obtain a target posture network, wherein the target posture network is used to determine the posture information of the target object in the image to be processed in combination with the feature extraction network.

[0145] In one possible implementation, the training module 63 is configured to: perform multiple rounds of iterative training on the initial posture network according to the feature tensor of the target object to obtain multiple alternative posture networks; and select a target posture network from the multiple alternative posture networks when the training end conditions are met, and the training end conditions include that the number of iterative training rounds of the initial posture network is greater than or equal to a first preset threshold, and / or that the learning rate of the current round of iterative training is less than a second preset threshold.

[0146] In one possible implementation, the initial posture network is iteratively trained for multiple rounds based on the feature tensor of the target object to obtain multiple alternative posture networks, including: for any round of iterative training of the initial posture network, the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result; the posture loss is determined based on a preset posture loss function, the posture prediction result, and the annotation information of the training data set, wherein the posture loss is used to indicate the detection error of the initial posture network on the posture information of the target object; based on the posture loss, the initial posture network is iteratively trained to obtain the alternative posture network corresponding to the current round of iterative training.

[0147] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result, including: inputting the feature tensor of the target object into the first network to obtain a posture prediction result, and the posture prediction result includes the rotation matrix of the target object predicted by the first network; the preset posture loss function includes a first loss function, and the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network. According to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, the posture loss is determined, including: inputting the rotation matrix of the target object predicted by the first network and the annotation information of the training data set corresponding to the target object into the first loss function to determine the posture loss.

[0148] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object. The feature tensor of the target object is input into the initial posture network to obtain a posture prediction result, including: inputting the feature tensor of the target object into the first network and the second network respectively to obtain a posture prediction result, the posture prediction result including the rotation matrix of the target object predicted by the first network and the posture angle of the target object predicted by the second network; the preset posture loss function includes the product of a first loss function and a second loss function, the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network, and the second loss function is used to indicate the detection error of the posture angle of the target object predicted by the second network. According to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, the posture loss is determined, including: inputting the rotation matrix of the target object predicted by the first network, the posture angle of the target object predicted by the second network, and the annotation information of the training data set corresponding to the target object into the preset posture loss function to determine the posture loss.

[0149] In one possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, the first network including a first network layer, a second network layer, and a third network layer; wherein the first network layer includes a convolution unit with a convolution kernel size of a first preset size, a normalization unit, and a nonlinear activation unit; the second network layer includes a depthwise separable convolution unit with a convolution kernel size of a second preset size, a normalization unit, and a nonlinear activation unit, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit.

[0150] In a possible implementation, the initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object; the first network includes a first network layer, a second network layer, and a third network layer, and the second network includes a fourth network layer, a fifth network layer, a sixth network layer, and a seventh network layer; wherein the first network layer includes a convolution unit with a convolution kernel size of a first preset size, a normalization unit, and a nonlinear activation unit; the second network layer includes a depthwise separable convolution unit with a convolution kernel size of a second preset size, a normalization unit, and a nonlinear activation unit, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit; the fourth network layer includes a convolution unit with a convolution kernel size of the first preset size, a normalization unit, and a nonlinear activation unit; the fifth network layer includes a depthwise separable convolution unit with a convolution kernel size of the second preset size, a normalization unit, and a nonlinear activation unit; the sixth network layer includes a unit convolution unit, a normalization unit, and a nonlinear activation unit; and the seventh network layer includes a unit convolution unit.

[0151] In one possible implementation, the acquisition module 61 is configured to: acquire a first sample set, the first sample set including sample images in which the posture angle of the target object is greater than a preset angle threshold; acquire a second sample set, the second sample set including sample images collected by multiple sensors in different scenarios; perform data set enhancement processing on the first sample set and / or the second sample set to obtain a third sample set; and determine a training data set based on the first sample set, the second sample set, and the third sample.

[0152] Figure 7 shows a block diagram of a posture estimation device according to an embodiment of the present disclosure. As shown in Figure 7, the device includes: a module 71 for acquiring an image to be processed, configured to acquire an image to be processed; a feature extraction module 72, configured to input the image to be processed into a preset feature extraction network to obtain a feature tensor of the target object in the image to be processed; a posture estimation module 73, configured to input the feature tensor of the target object in the image to be processed into a target posture network for processing, and determine a detection result of the image to be processed, wherein the detection result includes the posture information of the target object; wherein the target posture network is trained by the above-mentioned network training method for posture estimation.

[0153] In some embodiments, the functions or modules included in the apparatus provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment, and its specific implementation can refer to the description of the above method embodiment.

[0154] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0155] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0156] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0157] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0158] FIG7 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to FIG7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions that can be executed by the processing component 1922, such as an application. The application stored in the memory 1932 can include one or more modules, each of which corresponds to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0159] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0160] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0161] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0162] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0163] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0164] The computer program instructions for performing the operation of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer and partially on a remote computer, or executed entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0165] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0166] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0167] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0168] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0169] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0170] The above description of the various embodiments tends to emphasize the differences between the various embodiments, and reference can be made to the same or similar aspects thereof.

[0171] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0172] If the technical solution disclosed herein involves personal information, the product using the technical solution disclosed herein has clearly informed the individual of the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution disclosed herein involves sensitive personal information, the product using the technical solution disclosed herein has obtained the individual's separate consent before processing the sensitive personal information and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the individual has entered the personal information collection scope and that personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0173] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein. Industrial Applicability

[0174] In an embodiment of the present disclosure, the sample images in the acquired training data set can be input into a preset feature extraction network (e.g., a key point feature extraction network built into the driver monitoring system DMS) to train the initial posture network based on the feature tensor of the target object in the sample image output by the feature extraction network to obtain a target posture network. The target posture network trained in this way can be added to the already trained feature extraction network (e.g., a key point feature extraction network built into the driver monitoring system DMS) during network deployment in the inference phase without performing additional data preprocessing. The feature tensor output by the existing feature extraction network can be directly reused, which is beneficial to reducing the occupancy of computing resources and dependence on related algorithm libraries during deployment. Moreover, during the network training process, no feature extraction network-related training is required, which can improve training efficiency and further reduce the consumption of processor computing power resources and memory resources.

Claims

1. A network training method for posture estimation, comprising: Get the training dataset; Inputting the sample images in the training data set into a preset feature extraction network to determine the feature tensor of the target object in the sample image; The initial posture network is trained according to the feature tensor of the target object to obtain a target posture network, wherein the target posture network is used to determine the posture information of the target object in the image to be processed in combination with the feature extraction network.

2. The method according to claim 1, wherein: The step of training the initial posture network according to the feature tensor of the target object to obtain the target posture network includes: According to the feature tensor of the target object, the initial posture network is iteratively trained for multiple rounds to obtain multiple candidate posture networks; When the training termination condition is met, a target posture network is selected from a plurality of candidate posture networks, wherein the training termination condition includes that the number of iterative training rounds for the initial posture network is greater than or equal to a first preset threshold, and / or the learning rate of the current round of iterative training is less than a second preset threshold.

3. The method according to claim 2, wherein: According to the feature tensor of the target object, the initial posture network is iteratively trained for multiple rounds to obtain multiple candidate posture networks, including: For any round of iterative training of the initial posture network, the feature tensor of the target object is input into the initial posture network to obtain a posture prediction result; Determine a posture loss according to a preset posture loss function, the posture prediction result, and the annotation information of the training data set, wherein the posture loss is used to indicate a detection error of the initial posture network on the posture information of the target object; According to the posture loss, the initial posture network is iteratively trained to obtain an alternative posture network corresponding to the current round of iterative training.

4. The method according to claim 3, wherein: The initial posture network includes a first network for predicting a rotation matrix of a target object, and inputting a feature tensor of the target object into the initial posture network to obtain a posture prediction result includes: Inputting the feature tensor of the target object into a first network to obtain a posture prediction result, wherein the posture prediction result includes a rotation matrix of the target object predicted by the first network; The preset posture loss function includes a first loss function, and the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network. The posture loss is determined according to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, including: The rotation matrix of the target object predicted by the first network and the labeling information of the training data set corresponding to the target object are input into a first loss function to determine the posture loss.

5. The method according to claim 3 or 4, wherein: The initial posture network includes a first network for predicting a rotation matrix of the target object, and a second network for predicting a posture angle of the target object; The step of inputting the feature tensor of the target object into an initial posture network to obtain a posture prediction result includes: Inputting the feature tensor of the target object into the first network and the second network respectively to obtain a posture prediction result, wherein the posture prediction result includes a rotation matrix of the target object predicted by the first network and a posture angle of the target object predicted by the second network; The preset posture loss function includes the product of a first loss function and a second loss function, the first loss function is used to indicate the detection error of the rotation matrix of the target object predicted by the first network, and the second loss function is used to indicate the detection error of the posture angle of the target object predicted by the second network. According to the preset posture loss function, the posture prediction result, and the annotation information of the training data set, the posture loss is determined, including: The rotation matrix of the target object predicted by the first network, the posture angle of the target object predicted by the second network, and the labeling information of the training data set corresponding to the target object are input into the preset posture loss function to determine the posture loss.

6. The method according to any one of claims 1 to 5, wherein: The initial posture network includes a first network for predicting a rotation matrix of a target object, wherein the first network includes a first network layer, a second network layer, and a third network layer; Among them, the first network layer includes a convolution unit, a normalization unit, and a non-linear activation unit whose convolution kernel size is a first preset size; the second network layer includes a depth-separable convolution unit, a normalization unit, and a non-linear activation unit whose convolution kernel size is a second preset size, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit.

7. The method according to any one of claims 1 to 6, wherein: The initial posture network includes a first network for predicting the rotation matrix of the target object, and a second network for predicting the posture angle of the target object; the first network includes a first network layer, a second network layer, and a third network layer, and the second network includes a fourth network layer, a fifth network layer, a sixth network layer, and a seventh network layer; The first network layer includes a convolution unit, a normalization unit, and a nonlinear activation unit, whose convolution kernel size is a first preset size; the second network layer includes a depthwise separable convolution unit, a normalization unit, and a nonlinear activation unit, whose convolution kernel size is a second preset size, and the second preset size is larger than the first preset size; the third network layer includes a unit convolution unit; The fourth network layer includes a convolution unit whose convolution kernel size is the first preset size, a normalization unit, and a nonlinear activation unit; the fifth network layer includes a depth-separable convolution unit whose convolution kernel size is the second preset size, a normalization unit, and a nonlinear activation unit; the sixth network layer includes a unit convolution unit, a normalization unit, and a nonlinear activation unit; the seventh network layer includes a unit convolution unit.

8. The method according to any one of claims 1 to 7, wherein: The step of obtaining a training data set includes: Acquire a first sample set, the first sample set including sample images of which the posture angle of the target object is greater than a preset angle threshold; Acquire a second sample set, where the second sample set includes sample images collected by multiple sensors in different scenes; Performing data set enhancement processing on the first sample set and / or the second sample set to obtain a third sample set; A training data set is determined according to the first sample set, the second sample set, and the third sample.

9. A posture estimation method, the method comprising: Get the image to be processed; Inputting the image to be processed into a preset feature extraction network to obtain a feature tensor of a target object in the image to be processed; Inputting the feature tensor of the target object in the image to be processed into the target posture network for processing, and determining the detection result of the image to be processed, wherein the detection result includes the posture information of the target object; The target posture network is trained according to the network training method for posture estimation according to any one of claims 1 to 8.

10. A network training device for posture estimation, comprising: An acquisition module, configured to acquire a training data set; A determination module configured to input a sample image in the training data set into a preset feature extraction network to determine a feature tensor of a target object in the sample image; The training module is configured to train the initial posture network according to the feature tensor of the target object to obtain a target posture network, wherein the target posture network is used to determine the posture information of the target object in the image to be processed in combination with the feature extraction network.

11. A posture estimation device, comprising: A module for acquiring an image to be processed, configured to acquire an image to be processed; A feature extraction module is configured to input the image to be processed into a preset feature extraction network to obtain a feature tensor of a target object in the image to be processed; A posture estimation module is configured to input a feature tensor of a target object in the image to be processed into a target posture network for processing, and determine a detection result of the image to be processed, wherein the detection result includes posture information of the target object; Wherein, the target posture network is a network training for posture estimation according to any one of claims 1 to 8. It is obtained through training through practice methods.

12. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 9.

13. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

14. A computer program product comprising computer executable instructions, which when executed by a processor implement the method of any one of claims 1 to 9.

Images