Method for determining the posture of a human body and device for determining the posture of a human body

Abstract

This application provides a method and apparatus for determining the full-body posture of a human body. The method includes: first, acquiring head posture information, hand posture information, and foot pressure information of a target object in real time; then, inputting the head posture information, hand posture information, and foot pressure information into a deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based at least on posture images obtained from a depth camera. By using a deep learning model obtained in advance based at least on depth camera data, the accuracy of the determined full-body posture and full-body skeletal points is ensured to be high. This ensures that accurate full-body posture and full-body skeletal points can be obtained using only head posture information, hand posture information, and foot pressure information, thus ensuring the high practicality of the full-body posture determination process and solving the problem in existing technologies where the full-body posture cannot be predicted simply and accurately.

Description

Technical Field

[0001] This application relates to the field of human posture, and more specifically, to a method for determining the posture of the whole human body, a device for determining the posture, a computer-readable storage medium, a processor, and an electronic device. Background Technology

[0002] Currently, we mainly predict human posture by placing optical reflective points on the body, using inertial sensors, or pressure sensors. However, placing optical reflective points is expensive and troublesome, inertial sensors become increasingly noisy over time and have low practicality, and predicting the whole body posture by simply using pressure sensors has low accuracy, with prediction errors exceeding 10 centimeters.

[0003] Therefore, there is an urgent need for a simple and accurate method to predict the posture of the entire human body.

[0004] The information disclosed above in the background section is only intended to enhance the understanding of the background art of the art described herein. Therefore, the background art may contain certain information that does not constitute prior art known to those skilled in the art in this country. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, computer-readable storage medium, processor, and electronic device for determining the posture of the whole human body, so as to solve the problem that the prior art cannot predict the posture of the whole human body simply and accurately.

[0006] According to one aspect of the present invention, a method for determining the whole-body posture is provided. The method includes: acquiring head posture information, hand posture information, and foot pressure information of a target object in real time; inputting the head posture information, hand posture information, and foot pressure information into a deep learning model to obtain the whole-body posture and whole-body skeletal points of the target object, wherein the deep learning model is trained by machine learning based at least on posture images obtained from a depth camera.

[0007] Optionally, before inputting the head pose information, hand pose information, and foot pressure information into the deep learning model, the method further includes: establishing an initial deep learning model; acquiring multiple pose images of the target object using the depth camera, and extracting and analyzing the pose images to obtain historical poses corresponding to the pose images, wherein the historical poses include historical full-body poses and historical full-body skeletal points; acquiring historical foot pressure information, historical head pose information, and historical hand pose information corresponding to each pose image; and training the initial deep learning model based on multiple sets of historical foot pressure information, historical head pose information, historical hand pose information, and corresponding multiple sets of historical poses to obtain the deep learning model.

[0008] Optionally, obtaining historical foot pressure information, historical head posture information, and historical hand posture information corresponding to each of the posture images includes: obtaining the historical head posture information and the historical hand posture information corresponding to each of the posture images; obtaining multiple initial foot pressure information, wherein the initial foot pressure information is the foot pressure information corresponding to each of the posture images; performing a first preprocessing on the initial foot pressure information to obtain the historical foot pressure information, wherein the first preprocessing includes dataset augmentation processing.

[0009] Optionally, the initial deep learning model is trained based on multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures to obtain the deep learning model. This includes: generating a data matrix of multiple postures based on multiple sets of historical foot pressure information, multiple sets of historical head posture information, and multiple sets of historical hand posture information; and optimizing the initial deep learning model using a loss function and a backpropagation algorithm based on the differences between the data matrices and the corresponding historical postures to obtain the deep learning model.

[0010] Optionally, before training the initial deep learning model based on multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding multiple sets of historical postures to obtain the deep learning model, the method further includes: using timestamps to perform time synchronization processing on multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding multiple sets of historical postures.

[0011] Optionally, real-time acquisition of head posture information, hand posture information, and foot pressure information of the target object includes: controlling an augmented reality device or a virtual reality device to acquire the head posture information and hand posture information of the target object; acquiring the foot pressure information of the target object detected by a pressure sensor, wherein the spacing between the pressure sensing points of the pressure sensor is 1 cm.

[0012] According to another aspect of the present invention, a device for determining the full-body posture of a human body is also provided. The device includes a first acquisition unit and an input unit, wherein the first acquisition unit is used to acquire head posture information, hand posture information, and foot pressure information of a target object in real time; the input unit is used to input the head posture information, the hand posture information, and the foot pressure information into a deep learning model to obtain the full-body posture and full-body skeletal points of the target object, wherein the deep learning model is trained by machine learning based at least on posture images obtained from a depth camera.

[0013] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program is used to perform any of the methods described.

[0014] According to another aspect of the present invention, a processor is also provided, the processor being configured to run a program, wherein the program, when running, performs any of the methods described.

[0015] According to another aspect of the present invention, an electronic device is also provided, the electronic device including one or more processors, a memory, a display device, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any of the methods described.

[0016] In this embodiment of the invention, the method for determining the full-body posture first acquires the head posture information, hand posture information, and foot pressure information of the target object in real time; then, the head posture information, hand posture information, and foot pressure information are input into a deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based at least on posture images obtained from a depth camera. Compared to the problem of existing technologies that cannot easily and accurately predict the full-body posture of the human body, the method for determining the full-body posture of this application, by using the deep learning model obtained in advance (at least based on the depth camera) and then inputting the real-time acquired head posture information, hand posture information, and foot pressure data of the target object into the already obtained deep learning model, reverse-engineers the full-body posture and full-body skeletal points of the target object. This ensures high accuracy of the full-body posture and full-body skeletal points, and guarantees that accurate full-body posture and full-body skeletal points can be obtained using only the head posture information, hand posture information, and foot pressure information. This ensures high practicality of the full-body posture determination process and solves the problem of existing technologies that cannot easily and accurately predict the full-body posture of the human body. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 A schematic flowchart of a method for determining the full-body posture of a human body according to an embodiment of this application is shown;

[0019] Figure 2 A schematic diagram of a device for determining the full-body posture of a human body according to an embodiment of this application is shown. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be an intermediate element present. Furthermore, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element via a third element.

[0024] As mentioned in the background section, the prior art cannot easily and accurately predict the posture of the whole human body. In order to solve the above problem, in a typical embodiment of this application, a method for determining the posture of the whole human body, a device for determining the posture, a computer-readable storage medium, a processor, and an electronic device are provided.

[0025] According to an embodiment of this application, a method for determining the full-body posture of a human body is provided.

[0026] Figure 1 This is a flowchart of a method for determining the full-body posture according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0027] Step S101: Real-time acquisition of head posture information, hand posture information, and foot pressure information of the target object;

[0028] Step S102: Input the head posture information, hand posture information and foot pressure information into the deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based on the posture images obtained from the depth camera.

[0029] In the aforementioned method for determining full-body posture, firstly, the head posture information, hand posture information, and foot pressure information of the target object are acquired in real time; then, the head posture information, hand posture information, and foot pressure information are input into a deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained using machine learning based at least on posture images obtained from a depth camera. Compared to the problem of existing technologies failing to predict full-body posture simply and accurately, the method for determining full-body posture in this application, based at least on the deep learning model obtained from the depth camera, and further using the real-time acquired head posture information, hand posture information, and foot pressure data of the target object, inputs these data into the acquired deep learning model to infer the full-body posture and full-body skeletal points of the target object. This ensures high accuracy of the full-body posture and full-body skeletal points, and guarantees that accurate full-body posture and full-body skeletal points can be obtained using only the head posture information, hand posture information, and foot pressure information. This ensures high practicality of the full-body posture determination process and solves the problem of existing technologies failing to predict full-body posture simply and accurately.

[0030] According to a specific embodiment of this application, before inputting the head pose information, hand pose information, and foot pressure information into the deep learning model, the method further includes: establishing an initial deep learning model; acquiring multiple pose images of the target object using the depth camera, and extracting and analyzing the pose images to obtain historical poses corresponding to the pose images, wherein the historical poses include historical full-body poses and historical full-body skeletal points; acquiring historical foot pressure information, historical head pose information, and historical hand pose information corresponding to each pose image; and training the initial deep learning model based on multiple sets of historical foot pressure information, historical head pose information, historical hand pose information, and corresponding multiple sets of historical poses to obtain the deep learning model. By establishing the initial deep learning model, acquiring accurate pose images through the depth camera, processing the pose images to obtain the historical poses, and finally training the initial deep learning model with the acquired historical foot pressure information, historical head pose information, historical hand pose information, and corresponding historical poses, the accuracy of the deep learning model is ensured, further ensuring the accuracy of the whole-body pose and the whole-body skeletal points derived from the deep learning model.

[0031] In one specific embodiment, the image obtained by the depth camera is processed by a multi-layer convolutional neural network model to obtain the precise three-dimensional position / rotation angle (including X / Y / Z + rotation angle) of 32 skeletal points, and the output is a 32x7 matrix.

[0032] Specifically, the aforementioned depth cameras include Azure DK depth cameras.

[0033] To further ensure the high accuracy of the determined full-body postures, according to another specific embodiment of this application, historical foot pressure information, historical head posture information, and historical hand posture information corresponding to each of the aforementioned posture images are obtained. This includes: obtaining the historical head posture information and the historical hand posture information corresponding to each of the aforementioned posture images; obtaining multiple initial foot pressure information, wherein the initial foot pressure information is the foot pressure information corresponding to each of the aforementioned posture images; and performing a first preprocessing on the initial foot pressure information to obtain the historical foot pressure information, wherein the first preprocessing includes dataset augmentation processing. By performing the first preprocessing on the initial foot pressure data, the accuracy of the obtained historical foot pressure information is ensured to be high, further ensuring the high accuracy of the aforementioned deep learning model.

[0034] Specifically, the first preprocessing performed on the initial foot pressure information also includes the removal of edge values.

[0035] According to another specific embodiment of this application, the initial deep learning model is trained based on multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures to obtain the deep learning model. This includes: generating a data matrix of multiple postures based on multiple sets of historical foot pressure information, multiple sets of historical head posture information, and multiple sets of historical hand posture information; and optimizing the initial deep learning model using a loss function and a backpropagation algorithm based on the differences between the data matrix and the corresponding historical postures to obtain the deep learning model. By generating the data matrix using multiple sets of historical foot pressure information, multiple sets of historical head posture information, and multiple sets of historical hand posture information, and then optimizing the initial deep learning model using a loss function and a backpropagation algorithm by comparing the differences between the data matrix and the historical postures to obtain the deep learning model, the accuracy of the deep learning model is ensured to be high, further ensuring the high accuracy of the whole-body posture and the whole-body skeletal points derived from the deep learning model.

[0036] Specifically, the aforementioned historical foot pressure information is fed into a neural network model, which can be a Transformer (transformer algorithm model) or LSTM (Long Short Term Memory) network, or other algorithms. Through this neural network, a pressure feature matrix is ​​learned.

[0037] In one specific embodiment, a gradient descent algorithm is used to reduce the loss.

[0038] According to a specific embodiment of this application, before training the initial deep learning model based on multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures to obtain the deep learning model, the method further includes: using timestamps to perform time synchronization processing on the multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures. By using timestamps to perform time synchronization processing on the multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures, the accuracy of the subsequently obtained deep learning model is further ensured, and the accuracy of the whole-body posture and the whole-body skeletal points inferred from the deep learning model is further ensured.

[0039] According to another specific embodiment of this application, real-time acquisition of head posture information, hand posture information, and foot pressure information of a target object includes: controlling an augmented reality device or a virtual reality device to acquire the head posture information and hand posture information of the target object; acquiring the foot pressure information of the target object detected by a pressure sensor, wherein the spacing between the pressure sensing points of the pressure sensor is 1 cm. By acquiring the head posture information and hand posture information of the target object through the augmented reality device or the virtual reality device, and by acquiring the foot pressure information through the foot pressure sensor, the acquisition of the head posture information, hand posture information, and foot pressure information can be easily achieved, ensuring the high practicality of the process for determining the overall posture of the human body.

[0040] In one specific embodiment, the aforementioned head posture information and hand posture information are obtained through the VR (Virtual Reality) / AR (Augmented Reality) headset / controller via the API (Application Programming Interface) provided by the headset manufacturer. For example, SteamVR (360° room-type virtual reality) can obtain the real-time position of headsets such as HTC Vive (virtual reality head-mounted display) and Valve Index (virtual reality head-mounted display).

[0041] Specifically, it is not limited to obtaining the head posture information and hand posture information through the aforementioned augmented reality device or virtual reality device; other devices that can obtain the head posture information and hand posture information can also be selected according to the actual situation.

[0042] In one specific embodiment, the acquisition process of the head posture information, hand posture information and foot pressure information is fully automatic, which can achieve more than 100 frames per second, and up to 200Hz in actual use, with a delay of no more than 20ms.

[0043] Specifically, the aforementioned postures include sitting postures and sleeping postures, etc. The method for determining the above-mentioned full-body postures can be applied to Unity (a 3D interactive content creation and operation platform), Unreal Engine, or other 3D (3-Dimension) rendering engines and imported into 3D games in FBX (Film Box) format.

[0044] This application also provides a device for determining full-body posture. It should be noted that this device can be used to execute the method for determining full-body posture provided in this application. The following describes the device for determining full-body posture provided in this application.

[0045] Figure 2 This is a schematic diagram of a whole-body posture determination device according to an embodiment of this application. Figure 2 As shown, the device includes a first acquisition unit 10 and an input unit 20. The first acquisition unit 10 is used to acquire head posture information, hand posture information, and foot pressure information of the target object in real time. The input unit 20 is used to input the head posture information, hand posture information, and foot pressure information into a deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based on posture images obtained from a depth camera.

[0046] In the aforementioned full-body posture determination device, the first acquisition unit acquires the head posture information, hand posture information, and foot pressure information of the target object in real time; the input unit inputs the head posture information, hand posture information, and foot pressure information into a deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based at least on posture images obtained from a depth camera. Compared to the problem of existing technologies that cannot easily and accurately predict the full-body posture of the human body, the full-body posture determination device of this application, by using the deep learning model obtained in advance based at least on the depth camera, and then inputting the real-time acquired head posture information, hand posture information, and foot pressure data of the target object into the already obtained deep learning model, reverse-engineers the full-body posture and full-body skeletal points of the target object. This ensures high accuracy of the full-body posture and full-body skeletal points, and guarantees that accurate full-body posture and full-body skeletal points can be obtained using only the head posture information, hand posture information, and foot pressure information. This ensures high practicality of the full-body posture determination process and solves the problem of existing technologies that cannot easily and accurately predict the full-body posture of the human body.

[0047] According to a specific embodiment of this application, the above-mentioned device further includes an establishment unit, a second acquisition unit, a third acquisition unit, and a training unit. The establishment unit is used to establish an initial deep learning model before inputting the head pose information, hand pose information, and foot pressure information into the deep learning model. The second acquisition unit is used to acquire multiple pose images of the target object using the depth camera, and to extract and analyze the pose images to obtain historical poses corresponding to the pose images. The historical poses include historical full-body poses and historical full-body skeletal points. The third acquisition unit is used to acquire historical foot pressure information, historical head pose information, and historical hand pose information corresponding to each pose image. The training unit is used to train the initial deep learning model based on multiple sets of historical foot pressure information, historical head pose information, historical hand pose information, and corresponding multiple sets of historical poses to obtain the deep learning model. By establishing the initial deep learning model, acquiring accurate pose images through the depth camera, processing the pose images to obtain the historical poses, and finally training the initial deep learning model with the acquired historical foot pressure information, historical head pose information, historical hand pose information, and corresponding historical poses, the accuracy of the deep learning model is ensured, further ensuring the accuracy of the whole-body pose and the whole-body skeletal points derived from the deep learning model.

[0048] In one specific embodiment, the image obtained by the depth camera is processed by a multi-layer convolutional neural network model to obtain the precise three-dimensional position / rotation angle (including X / Y / Z + rotation angle) of 32 skeletal points, and the output is a 32x7 matrix.

[0049] Specifically, the aforementioned depth cameras include Azure DK depth cameras.

[0050] To further ensure the high accuracy of the determined full-body postures, according to another specific embodiment of this application, the third acquisition unit includes a first acquisition module, a second acquisition module, and a processing module. The first acquisition module acquires historical head posture information and historical hand posture information corresponding to each of the aforementioned posture images. The second acquisition module acquires multiple initial foot pressure information, which are the foot pressure information corresponding to each of the aforementioned posture images. The processing module performs a first preprocessing on the initial foot pressure information to obtain the historical foot pressure information. The first preprocessing includes dataset augmentation. By performing the first preprocessing on the initial foot pressure data, the accuracy of the obtained historical foot pressure information is ensured to be high, further guaranteeing the high accuracy of the deep learning model.

[0051] Specifically, the first preprocessing performed on the initial foot pressure information also includes the removal of edge values.

[0052] According to another specific embodiment of this application, the training unit includes a generation module and an optimization module. The generation module generates a data matrix of multiple postures based on multiple historical foot pressure information, multiple historical head posture information, and multiple historical hand posture information. The optimization module optimizes the initial deep learning model using a loss function and a backpropagation algorithm based on the differences between the data matrix and the corresponding historical postures to obtain the deep learning model. By generating the data matrix using multiple historical foot pressure information, multiple historical head posture information, and multiple historical hand posture information, and then optimizing the initial deep learning model using a loss function and a backpropagation algorithm by comparing the differences between the data matrix and the historical postures to obtain the deep learning model, the accuracy of the deep learning model is ensured to be high, further ensuring the high accuracy of the whole-body posture and the whole-body skeletal points derived from the deep learning model.

[0053] Specifically, the aforementioned historical foot pressure information is fed into a neural network model, which can be a Transformer or LSTM (Long Short Term Memory) network, or other algorithms. This neural network learns to obtain a pressure feature matrix.

[0054] In one specific embodiment, a gradient descent algorithm is used to reduce the loss.

[0055] According to a specific embodiment of this application, the device further includes a processing unit. This processing unit is used to perform time synchronization processing on the multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures before training the initial deep learning model based on multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures. By using timestamps to perform time synchronization processing on the multiple sets of historical foot pressure information, historical head posture information, historical hand posture information, and corresponding sets of historical postures, the accuracy of the subsequently obtained deep learning model is further ensured, and the accuracy of the whole-body posture and whole-body skeletal points inferred from the deep learning model is further ensured.

[0056] According to another specific embodiment of this application, the first acquisition unit includes a third acquisition module and a fourth acquisition module. The third acquisition module controls an augmented reality device or a virtual reality device to acquire the head posture information and hand posture information of the target object. The fourth acquisition module acquires the foot pressure information of the target object detected by a pressure sensor, wherein the spacing between the pressure sensing points of the pressure sensor is 1 cm. By acquiring the head posture information and hand posture information of the target object through the augmented reality device or the virtual reality device, and by acquiring the foot pressure information through the foot pressure sensor, the acquisition of the head posture information, hand posture information, and foot pressure information can be easily achieved, ensuring the high practicality of the process for determining the overall posture of the human body.

[0057] In one specific embodiment, the aforementioned head posture information and hand posture information are obtained through the VR (Virtual Reality) / AR (Augmented Reality) headset / controller via the API (Application Programming Interface) provided by the headset manufacturer. For example, SteamVR (360° room-type virtual reality) can obtain the real-time position of headsets such as HTC Vive (virtual reality head-mounted display) and Valve Index (virtual reality head-mounted display).

[0058] Specifically, it is not limited to obtaining the head posture information and hand posture information through the aforementioned augmented reality device or virtual reality device; other devices that can obtain the head posture information and hand posture information can also be selected according to the actual situation.

[0059] In one specific embodiment, the acquisition process of the head posture information, hand posture information and foot pressure information is fully automatic, which can achieve more than 100 frames per second, and up to 200Hz in actual use, with a delay of no more than 20ms.

[0060] Specifically, the aforementioned postures include sitting postures and sleeping postures, etc. The method for determining the above-mentioned full-body postures can be applied to Unity (a 3D interactive content creation and operation platform), Unreal Engine, or other 3D (3-Dimension) rendering engines and imported into 3D games in FBX (Film Box) format.

[0061] The aforementioned whole-body posture determination device includes a processor and a memory. The aforementioned first acquisition unit and the aforementioned input unit are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0062] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the problem of the inability of existing technologies to predict the full-body posture of the human body simply and accurately.

[0063] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0064] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described method for determining the whole-body posture.

[0065] This invention provides a processor for running a program, wherein the program executes the method for determining the whole-body posture.

[0066] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0067] Step S101: Real-time acquisition of head posture information, hand posture information, and foot pressure information of the target object;

[0068] Step S102: Input the head posture information, hand posture information and foot pressure information into the deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based on the posture images obtained from the depth camera.

[0069] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0070] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0071] Step S101: Real-time acquisition of head posture information, hand posture information, and foot pressure information of the target object;

[0072] Step S102: Input the head posture information, hand posture information and foot pressure information into the deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based on the posture images obtained from the depth camera.

[0073] According to another typical embodiment of this application, an electronic device is also provided, the electronic device including one or more processors, a memory, a display device, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any of the above-described methods.

[0074] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0076] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0078] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0079] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0080] 1) In the method for determining the full-body posture described in this application, firstly, the head posture information, hand posture information, and foot pressure information of the target object are acquired in real time; then, the head posture information, hand posture information, and foot pressure information are input into a deep learning model to obtain the full-body posture and full-body skeletal points of the target object. The deep learning model is trained by machine learning based at least on posture images obtained from a depth camera. Compared to the problem in existing technologies where the full-body posture cannot be predicted simply and accurately, the method for determining the full-body posture described in this application, by using the deep learning model obtained in advance based at least on the depth camera, and then inputting the head posture information, hand posture information, and foot pressure data of the target object acquired in real time into the acquired deep learning model, reverse-engineers the full-body posture and full-body skeletal points of the target object. This ensures high accuracy of the full-body posture and full-body skeletal points, and guarantees that accurate full-body posture and full-body skeletal points can be obtained using only the head posture information, hand posture information, and foot pressure information. This ensures high practicality of the full-body posture determination process and solves the problem in existing technologies where the full-body posture cannot be predicted simply and accurately.

[0081] 2) In the above-mentioned whole-body posture determination device of this application, the head posture information, hand posture information and foot pressure information of the target object are acquired in real time by the first acquisition unit; the head posture information, hand posture information and foot pressure information are input into the deep learning model by the input unit to obtain the whole-body posture and whole-body skeletal points of the target object, wherein the deep learning model is trained by machine learning based at least on the posture images obtained by the depth camera. Compared to existing technologies that cannot easily and accurately predict the full-body posture of the human body, the full-body posture determination device of this application, by using the deep learning model obtained in advance from the depth camera, and then using the head posture information, hand posture information, and foot pressure data of the target object acquired in real time, inputs the obtained deep learning model to infer the full-body posture and the full-body skeletal points of the target object. This ensures high accuracy of the full-body posture and the full-body skeletal points, and ensures that accurate full-body posture and the full-body skeletal points can be obtained using only the head posture information, hand posture information, and foot pressure information. This ensures high practicality of the full-body posture determination process and solves the problem of existing technologies that cannot easily and accurately predict the full-body posture of the human body.

[0082] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of determining a human whole body posture, characterized in that, The method comprises: Real-time acquisition of head posture information, hand posture information and foot pressure information of a target object, the posture including sitting posture and sleeping posture; Establishing an initial deep learning model; Using a depth camera to acquire a plurality of posture images of the target object, and extracting and analyzing the posture images to obtain historical postures corresponding to the posture images, the historical postures including historical full-body postures and historical full-body skeleton points; Acquiring historical foot pressure information, historical head posture information and historical hand posture information corresponding to each of the posture images, including: acquiring the historical head posture information and the historical hand posture information corresponding to each of the posture images; acquiring a plurality of initial foot pressure information, the initial foot pressure information being the foot pressure information corresponding to each of the posture images; first preprocessing the initial foot pressure information to obtain the historical foot pressure information, the first preprocessing including data set enhancement processing; Training the initial deep learning model according to a plurality of sets of historical foot pressure information, historical head posture information, historical hand posture information and corresponding historical postures to obtain the deep learning model, including: generating a plurality of posture data matrices according to a plurality of historical foot pressure information, a plurality of historical head posture information and a plurality of historical hand posture information; using a loss function and a back propagation algorithm to optimize the initial deep learning model according to the differences between the data matrices and the corresponding historical postures to obtain the deep learning model; Inputting the head posture information, the hand posture information and the foot pressure information into the deep learning model to obtain the full-body posture and the full-body skeleton point of the target object, wherein the deep learning model is trained by machine learning at least according to the posture images obtained by the depth camera.

2. The method of claim 1, wherein, Before training the initial deep learning model according to a plurality of sets of historical foot pressure information, historical head posture information, historical hand posture information and corresponding historical postures to obtain the deep learning model, the method further comprises: Using a timestamp to perform time synchronization processing on a plurality of sets of historical foot pressure information, historical head posture information, historical hand posture information and corresponding historical postures.

3. The method of claim 1, wherein, Real-time acquisition of head posture information, hand posture information and foot pressure information of a target object, comprising: Controlling an augmented reality device or a virtual reality device to acquire the head posture information and the hand posture information of the target object; Acquiring the foot pressure information of the target object detected by a pressure sensor, the distance between the pressure sensing points of the pressure sensor being 1 cm.

4. An apparatus for determining a posture of a human body, characterized by comprising: The device comprises: A first acquisition unit configured to acquire in real time head posture information, hand posture information and foot pressure information of a target object, the posture including sitting posture and sleeping posture; The input unit is configured to input the head posture information, the hand posture information, and the foot pressure information into a deep learning model to obtain a full-body posture and full-body skeleton points of the target object, wherein the deep learning model is trained by machine learning according to posture images obtained by a depth camera. The establishing unit is configured to establish an initial deep learning model before the head posture information, the hand posture information, and the foot pressure information are input into the deep learning model. The second acquisition unit is configured to acquire a plurality of posture images of the target object by using the depth camera, and to extract and analyze the posture images to obtain historical postures corresponding to the posture images, wherein the historical postures include historical full-body postures and historical full-body skeleton points. The third acquisition unit is configured to acquire historical foot pressure information, historical head posture information, and historical hand posture information corresponding to each of the posture images. The training unit is configured to train the initial deep learning model according to a plurality of sets of the historical foot pressure information, the historical head posture information, the historical hand posture information, and a plurality of sets of the historical postures corresponding thereto, to obtain the deep learning model. The training unit includes a generation module and an optimization module. The generation module is configured to generate a plurality of posture data matrices according to a plurality of sets of the historical foot pressure information, a plurality of sets of the historical head posture information, and a plurality of sets of the historical hand posture information. The optimization module is configured to optimize the initial deep learning model according to differences between the data matrices and the historical postures corresponding thereto by using a loss function and a back propagation algorithm, to obtain the deep learning model. The third acquisition unit includes a first acquisition module, a second acquisition module, and a processing module. The first acquisition module is configured to acquire the historical head posture information and the historical hand posture information corresponding to each of the posture images. The second acquisition module is configured to acquire a plurality of initial foot pressure information, wherein the initial foot pressure information is the foot pressure information corresponding to each of the posture images. The processing module is configured to perform first preprocessing on the initial foot pressure information to obtain the historical foot pressure information, wherein the first preprocessing includes data set enhancement processing.

5. A computer readable storage medium, characterized in that, The computer readable storage medium includes a stored program, wherein the program executes the method of any one of claims 1 to 3.

6. A processor, comprising: The processor is configured to run a program, wherein the program executes the method of any one of claims 1 to 3 when running.

7. An electronic device, comprising: The processor is configured to run a program, wherein the program executes the method of any one of claims 1 to 3 when running. The processor is configured to run a program, wherein the program executes the method of any one of claims 1 to 3 when running. The processor is configured to run a program, wherein the program executes the method of any one of claims 1 to 3 when running.

Images